What’s New in Red Hat OpenShift Origin 3.

10

OpenShift Commons Briefing

July 2018
OCP 3.10 - The Efficient Cluster
● Resource Management
● Descheduler (tech preview), CPU Manager, Ephemeral Storage,
HugePages
● Resilience
● Node Problem Detector, HA egress pods with DNS
● Workload Diversity
● Device Manager, Windows Containers (dev preview)
● Installation Automation
● TLS node bootstraping, static pods
● Security
● Etcd cipher coverage, Shared PID namespace options, more secured
router
 Self-Service / UX
Feature(s): OpenShift Automation (Ansible) Broker

What’s New for 3.10:

● Remove etcd from Automation Broker and move to using CRDs

● Broker will now use CRDs instead of a local etcd instance

● Make serviceInstance details available to the playbook

● Exposes the details at runtime of who provisioned a service to the provision and deprovision playbooks
● Such as OpenShift cluster dns suffix, username, namespace, ServiceInstance id

● Enhance error messages, so when a provision request fails the error is preserved and displayed to end user in
web console
● Allows APB to return custom error messages that gets surfaced by service catalog if a provisioning operation fails
● Eases troubleshooting and improves customer experience

3
 AWS AMAZON WEB SERVICES
Service
Broker
Service Broker New AWS Services:

Kinesis Data Streams

Key Management Service (KMS)

Lex

Polly

Rekognition

Translate (requires Preview registration)

SageMaker*

Additional RDS engines:

Aurora*, MariaDB, & PostgreSQL

* Coming soon!
 Self-Service / UX

Feature(s): Improved search within

catalog

Description: Show “top 5” results

How it Works:
● Weighting is given based on where the
match is found
● Factors include: title, description,
tagging
 Self-Service / UX

Feature(s): User chooses route for

application

Description: Need better way to show routes

for app

How it Works:
● Indication that there are multiple routes
● Annotate route that you’d like to be
primary

console.alpha.openshift.io/overview-app-route: ‘true’
 Self-Service / UX

Feature(s): Create generic secrets

Description: Allow users a way to create

opaque secrets
How it Works:
● User can already create secrets,
expanding to opaque
● Behaves like creating config maps
 Self-Service / UX

Feature(s): Service Catalog CLI $ svcat provision postgresql-instance --class

rh-postgresql-apb --plan dev --params-json
'{"postgresql_database":"admin","postgresql_password":"
admin","postgresql_user":"admin","postgresql_version":"
Description: Provision, bind services 9.6"}' -n szh-project
Name: postgresql-instance
from command line Namespace: szh-project
Status:
How it Works: Class: rh-postgresql-apb
Plan: dev
● Full set of commands to list,
Parameters:
describe, provision/deprovision and postgresql_database: admin
postgresql_password: admin
bind/unbind postgresql_user: admin
● Based on contribution from Azure postgresql_version: "9.6"

● Separate CLI as part of RPM

 Self-Service / UX

Miscellaneous Service Catalog

● Rename bind credential secret keys

● Improvements in reconciliation process, optimizations and removed false
messages (failed to provision)
● Flexible secret management (add, remove, change)
 DevExp / Builds

Feature(s): Jenkins items

● Sync removal of build jobs - this allows for

cleanup of old/stale jobs

● Jenkins updated to 2.107.3-1.1

● Update in Jenkins build agent (slave) images
- Node.js 8
- Maven 3.5
 Dev Tools - Local Dev

CDK 3.4:
● OpenShift Container Platform v3.9.14
● Image caching is enabled by-default
● HyperV users can assign a static IP to CDK
● Hostfolder mount using SSHFS (Technical preview)
● Uses overlay as the default storage driver

Minishift 1.21 / CDK 3.5 : 17-JUL-2018

● Native hypervisor (Hyper-V/xhyve/KVM) or VirtualBox
● Run CDK against an existing RHEL7 host.
● SSHFS is default technology for hostfolder share
● Local DNS server to reduce dependency on nip.io.
● Users will be able to use OCP 3.10
 Operator SDK

Feature(s): Dev tools to build Kubernetes

Embed unique
applications
operational knowledge

Description: Help customers/ISVs build and publish

Kubernetes applications that run like cloud services,
anywhere OpenShift runs

How it Works: Package and install

on OCP clusters
● Includes all scaffolding code
● Only need to build the logic specific to app
● Tool to publish and use on multiple clusters
● Supports Helm chart, Ansible PB or Go code
 Container Orchestration
Red Hat Contributing Projects:
Feature(s): Kubernetes Upstream Red Hat
Blog and Commons Webinar
Description: OpenShift 3.10 brings
enhancements in how efficiently you can
leverage the resources available from the
nodes across the cluster. From ephemeral
storage, CPU, memory pages, IP addresses,
and other resources available to the cluster,
OpenShift 3.10 more efficiently brings nodes
into the cluster and exposes its resources to
application services.
● API Aggregation
● CronJobs stabilizing
● Control API access from nodes
● PSP stabilizing
● Configurable pod resolv.conf
● Kubelet ComponentConfig API
● Mount namespace propagation
● PV handling with deleted pods and
orphaned binds
● Ephemeral Storage Handling
● CRD subresource and categories
● Container Storage Interface
● Kubectl extension handling
 CPU Manager Policy

Performance Pods # cat /etc/origin/node/node-config.yaml

...
Pod spec:

kubeletArguments: resources:
Feature(s): HugePages, CPU Manager, Device Manager ... requests:
feature-gates: cpu: 1
- CPUManager=true memory: 256Mi
cpu-manager-policy: limits:
- static cpu: 1
Description: We spoke about Device Manager here. CPU cpu-manager-reconcile-period: memory: 256Mi
Manager Policy allows you to tell kube that your workload - 5s
kube-reserved:
ubelet
requires an affinity to a CPU core. Maybe your workload - cpu=500m
device
needs CPU cache affinity and can’t handle being bounced Result: manager
around to different CPU cores on the node via normal fair
# oc exec pod-name -- cat /sys/fs/cgroup/cpuset/cpuset.cpus
share scheduling on linux. HugePages allows you to request 2
# oc exec pod-name -- grep ^Cpus_allowed_list /proc/1/status
that your workload consume a specific amount of Cpus_allowed_list: 2
HugePages.
HugePages

# cat /etc/origin/node/node-config.yaml # cat /etc/origin/master/master-config.yaml

How it Works: CPU manager is a flag on the kubelet that ... ...
has the option of none or static. Static will cause kubeletArguments: kubernetesMasterConfig:
... apiServerArguments:
guaranteed QoS pod access to exclusive CPU cores on the feature-gates: ...
- HugePages=true feature-gates:
node. HugePages is a flag you set to true on the master - HugePages=true
and kubelet. The nodes will then be able to tell if any
Pod spec:
HugePages are available and workloads can request them
resources:
via the pod definition. limits: Both the variable name and
hugepages-2Mi: 100Mi value are configurable.
 Tech
Node Preview
Feature(s): Node Problem Detector
Description: Daemon that runs on each node as a Problem Daemons:
daemonSet. The daemon tries to make the cluster
aware of node level faults that should make the node ● Kernel Monitor: monitors kernel log via
ubeletproblems according to
journald and reports
not schedulable. device
regex patterns manager

How it Works: When you start the node problem ● AbrtAdaptor: monitors the node for kernel
detector you tell it a port to broadcast the issues it problems and application crashes from journald
find over. The detector allows you to load
sub-daemons to do the data collection. There are 3 ● CustomerPluginMonitor: allows you to test
for any condition and exit on a 0 or 1 should
as of today. Issues found by the problem daemon you condition not be met.
can be classified as “NodeCondition” which means
stop node scheduling or “Event” which are only
informative.
 Tech
Node Preview
1. Master: /etc/origin/master/master-config.yaml
Feature(s): Protection of Local Ephemeral Storage kubernetesMasterConfig:
apiServerArguments:
feature-gates:
Description: Control the usage of local ephemeral - LocalStorageCapacityIsolation=true
storage feature on the nodes in order to prevent controllerArguments:
feature-gates:
users from exhausting all node local storage (logs, - LocalStorageCapacityIsolation=true
ubelet
empty dirs, copy on write layer) with their pods and device
manager
abusing other pods that happen to be on the same 2. Node: /etc/origin/node/node-config.yaml
kubeletArguments:
node. feature-gates:
- LocalStorageCapacityIsolation=true

How it Works: After turning on

3.) Launch pods with the following in their deploymentConfig
LocalStorageCapacityIsolation, pods submitted use
resources:
the limit and requested fields. Violations will result requests:
in an evicted pod. ephemeral-storage: 500Mi
limits:
Limit: ephemeral storage request when scheduling a container to a ephemeral-storage: 1Gi
node, then fences off the requested ephemeral storage on the chosen
node for the use of the container.
Request: provides a hard limit on the ephemeral storage that can be
allocated across all the processes in a container
 Tech
Node Preview
Feature(s): Descheduler
Description: Due to the fact a scheduler’s view of a
cluster is at a single point in time, a overall cluster’s
balance may become skewed from taints and
tolerations, evictions, affinities, and other life cycle
reasons such as node maintenance or new node
additions. As a result, you can have some nodes
become under or over utilized.
How it Works: A descheduler is a job running in a
pod that runs in the kube-system project. This
descheduler finds pods based on its policy and
evicts them in order to give them back to the
scheduler for replacement on the cluster. It does
not target static pods, those with high QoS,
daemonSets, or those with local storage.
Available Policies:
● RemoveDuplicates: if this policy is set the
descheduler looks for pods that are apart of the same
replicaSet or deployment that happen to have been
placed on the same node. It evicts the duplicates in
the hope the scheduler will place them on a different
ubelet
node. device
manager
● LowNodeUtilization: finds nodes that are under the
CPU, MEM, and # of pod thresholds you have set, it
will evict pods from other nodes in the hopes the
scheduler places the pods on these under utilized
nodes. There is also a setting to only trigger this if you
have more than X number of under utilized nodes.
● RemovePodsViolatingInterPodAntiAffinity and
RemovePodsViolatingNodeAffinity: re-evaluates the
pods that might have been forced to break their affinity
rules and evicts them for another chance to be places
on nodes that comply to their affinity or anti-affinity.
 Dev
Node Preview
Providing in the developer preview:

1.) Powershell script to satisfy container prerequisites on Windows

Feature(s): Windows Containers Server
2.) Installation process that allows you to install on one to many
Description: Be able to run Windows containers on nodes without deploying an overlay network
3.) Ansible playbooks to deploy and configure an experimental OVN
Windows Server 1709, 1803, and 2019 within a network on the OpenShift cluster
4.) Ansible playbooks to deploy and configure an experimental OVN
OpenShift cluster. network from CloudBaseubelet
on Windows Server. And to then
connect that Windows node to the OpenShift cluster
device
manager
How it Works: Join partnership between Microsoft
Features in the first drop:
and Red Hat. Microsoft will distribute and support,
through our joint co-located support process, the 1.) kubelet and pre-reqs (docker, networking plugins, etc)
2.) Join Windows node to OpenShift cluster
kubelet, configuration/installation, and networking 3.) Allow Windows access to certain projects (nodeSelector or taints
components that need to be installed on Windows. & tolerations)
4.) Work with templates in the Service Catalog
Red Hat will support the interaction with those 5.) Attach static storage to the container
components with the OpenShift cluster. 6.) Scale up and down the Windows container
7.) DNS resolvable URL for service to route object
Customers and partners can sign up for the 8.) East/west network connectivity to Linux pods
developer preview program here. The program will 9.) Delete Windows Container

start within the next 7 days. It has been delayed

due to technical difficulties. Video of it WORKING!!!
 Registry
Feature(s): Expose registry metrics with
OpenShift auth

Description: Registry metrics endpoint now

protected by built-in OpenShift auth

How it works:
● Registry provides an endpoint for
Prometheus metrics
● Route must be enabled
● Users with the appropriate role can access
metrics using their openshift credentials
● An admin defined shared secret can still be
used to access the metrics as well
 Installation
Feature(s): Run control plane as static pod

Description: Migrate control plane to static pods to leverage self-management of cluster

components and minimize direct host management

How it Works:
● In 3.10 and newer, control plane components (etcd, API, and controller manager) will now move to
running as static pods
● Goal is to reduce node level configuration in preparation for automated cluster configuration on immutable
infrastructure
● Unified control plane deployment methods across Atomic Host and RHEL; everything runs atop the kubelet.
● The standard upgrade process will migrate existing clusters automatically
 Installation
Feature(s): Bootstrapped Node Configuration

Description: Node configuration is now managed Image Reference: https://medium.com/@toddrosner/kubernetes-tls-bootstrapping-cf203776abc7

via API objects and synchronized to nodes

How it Works:
● In 3.10 and newer, all members of the [nodes] inventory group must be assigned an openshift_node_group_name (value is
used to select the configmap that configures each node)
● By default, there are five configmaps created: node-config-master, node-config-infra, node-config-compute,
node-config-master-infra, & node-config-all-in-one
● Last two place a node into multiple roles
● Note: configmaps are the authoritative definition of node labels; the old openshift_node_labels value is effectively ignored.
● If you want to deviate from default configuration, you must define the entire openshift_node_group dictionary in your inventory.
When using an INI based inventory it must be translated into a Python dictionary.
● The upgrade process will now block until you have the required configmaps in the openshift-node namespace
● Either accept the defaults or define openshift_node_groups to meet your needs, then run
playbooks/openshift-master/openshift_node_group.yml to create the configmaps
● Review the configmaps carefully to ensure that all desired configuration items are set then restart the upgrade
● Changes to these configmaps will propagate to all nodes within 5 minutes overwriting /etc/origin/node/node-config.yaml
 Networking
Feature(s): HA Setup For Egress Pods
NAMESPACE A

Description: In the first z-stream release of 3.10, egress

pods can have HA failover across secondary cluster POD POD POD POD
nodes in the event the primary node goes down.
EGRESS EGRESS
How it works: Namespaces are now allowed to have IP 1 NODE 1 IP 2 NODE 2

multiple egress IPs specified, hosted on different nodes,

so that if the primary node fails the egress IP switches
from its primary to secondary egress IP being hosted on EXTERNAL
another node. When the original IP eventually comes SERVICE
back, then nodes will switch back to using the original Whitelist: IP1, IP2
egress IP. The switchover currently takes ≤7 seconds
for a node to notice that an egress node has gone down
(potentially configurable in a later version).
 Networking
Feature(s): Allow DNS names for ...
egress routers - name: EGRESS_DESTINATION
value: |
80 tcp my.example.com
Description: The egress router can 8080 tcp 5.6.7.8 80
now refer to an external service, with a 8443 tcp your.example.com 443
potentially unstable IP address, by its 13.14.15.16
...
hostname.

POD
How it works: The OpenShift egress
router runs a service that redirects EGRESS EXTERNAL
egress pod traffic to one or more EGRESS ROUTER
POD SERVICE POD SERVICE
specified remote servers, using a INTERNAL-IP:8080 IP1
Whitelist: IP1
pre-defined source IP address that can
be whitelisted on the remote server. Its POD NODE
IP1
EGRESS_DESTINATION can now
specify the remote sever by FQDN.
 Networking
Feature(s): Document and test a supported way of How it works:
expanding the serviceNetwork
1. Update the master-config.yaml to change the
Description: Provide a supported way of growing the serviceNetworkCIDR to 172.30.0.0/16
service network address range in a multi-node 2. Delete the default clusternetwork object on the
environment to a larger address space. master:
# oc delete clusternetwork
For example: default
3. Restart the master API service and the
serviceNetworkCIDR: 172.30.0.0/24
controller service
172.30.0.0/16 4. Update the ansible inventory file to match the
change in (1) and redeploy the cluster
Note: This DOES NOT cover migration to a different 5. Evacuate the node one by one and restart the
range, JUST the increase of an existing range. iptables and atomic-openshift-node services
 Security
Feature(s) : Specify whitelist cipher suite for etcd

Description: Users now have the ability to optionally

whitelist cipher suites for use with etcd in order to
meet security policies.

How it Works:
● Configure etcd to add --cipher-suites flag with
the desired cipher suite
● Restart etcd, apiserver, controllers, etc
● TLS handshake fails when client hello is
requested with invalid cipher suites.
● If empty, Go auto-populates the list.
 Tech
Security Preview

Feature(s) : Control Sharing the PID namespace between containers

Description: Use this feature to configure cooperating containers in a pod, such as a
log handler sidecar container, or to troubleshoot container images that don’t include
debugging utilities like a shell.

How it Works:
● The feature gate PodShareProcessNamespace is set to false by default
● Set 'feature-gates=PodShareProcessNamespace=true'
in apiserver, controllers and kubelet
● Restart apiserver, controller and node service
● Create a pod with spec "shareProcessNamespace: true"
● oc create -f <pod spec file>

Caveats: When the pid namespace is shared between containers

● Sidecar containers are not isolated
● Environment variables are now visible to all other processes
● Any "kill all" semantics used within the process are now broken
● Exec processes from other containers will now show up pods/share-process-namespace.yaml
 Security
Feature(s) : Router Service Account no longer needs access to secrets

Description: The router service account no longer needs EXTERNAL

permission to read all secrets. This improves security, as TRAFFIC

previously, if the router were compromised it could then

ROUTER
read all of the most sensitive data in the cluster.

How it Works: SERVICE

INTERNAL
TRAFFIC

● When you create an ingress object, a corresponding

route object is created.
POD POD POD
● If an ingress object is modified, a changed secret
should take effect soon after
● If an ingress object is deleted, a route that was
created for it will be deleted
 Dev
Storage Preview

Feature(s): Container Storage Interface (CSI) How it Works

Description: Introduce CSI sub-system as tech • Create a new project where the CSI components will
preview in 3.10 run and a new service account that will run the
components
• Create the Deployment with the external CSI attacher
• External Attacher and provisioner and DaemonSet with the CSI driver
• Create a StorageClass for the new storage entity
• External Provisioner • Create a PVC with the new StorageClass
• Driver registrar
• See:
• CSI Drivers shipped: None (use
https://github.com/openshift/openshift-docs/blob/maste
external/upstream) r/install_config/persistent_storage/persistent_storage_
csi.adoc
 Tech
Storage Preview

Feature(s): New Storage Provisioners How it Works

• Use OpenShift Ansible installer
Description: New Storage Provisioners (external
openshift_provisioners role
provisioners) added as Tech Preview with 3.10
• Set the provisioner to be installed and started as true
<After the provisioner install and startup is completed>
• CephFS
• Create a Storage Class for the storage entity
• Create a pod with a PVC/claim with the Storage Class
 RHEL 7.5 Highlights

OpenShift Container Platform 3.10 is supported

on RHEL 7.4, 7.5 and Atomic Host 7.5+

Containers / Atomic
● Docker 1.13
● Docker-latest deprecation
● RPM-OSTree package overrides
Storage
Security
● Unprivileged mount namespace
● KASLR full support and enabled by default.
● Ansible remediation for OpenSCAP
● Improved SELinux labeling for cgroups
(cgroup_seclabel)
● Atomic Host deprecation notice, as Red Hat
CoreOS will be the future immutable host option.
○ Atomic supported in 3.10 & 3.11
● Virtual data optimizer (VDO) for dm-level dedupe
and compression.
● OverlayFS by default for new installs (overlay2)
○ Ensure ftype=1 for 7.3 and earlier
● Devicemapper continues to be supported and
available for edge cases around POSIX
● LVM snapshots integrated with boot loader (boom)
 CRI-O v1.10

Feature(s): CRI-O v1.10

Improvements include:
Description: CRI-O is an OCI compliant
implementation of the Kubernetes Container ● crictl CLI for debugging and troubleshooting
Runtime Interface. By design it provides only the ● Podman for image tagging & management
runtime capabilities needed by the kubelet. CRI-O is ● Installer integration & fresh install time
designed to be part of Kubernetes and evolve in decision: openshift_use_crio=True
lock-step with the platform. ● Not available for existing cluster upgrades

CRI-O brings:
CNI Networking RunC
Kubelet
● A minimal and secure architecture Storage Image
● Excellent scale and performance
● Ability to run any OCI / Docker image
● Familiar operational tooling and commands
Questions