Scaling DevOps with Puppet and Perforce

Christoph Leithner
Founder
CELIX Hard- & Software
VertriebsgmbH
http://www.celix.at

Logo area

Agenda
What is DevOps?
IT Automation with Puppet
Scaling DevOps with Perforce
Agile Flow of Change for Puppet Modules

What is DevOps?

What is DevOps?

DevOps
DevOps

Technology
Operations
Source: http://commons.wikimedia.org/wiki/File:Devops.png by Rajiv Pan

DevOps Areas - codified by Patrick Debois

Area 3: Embed project knowledge into operations
Area 1: Extend delivery
to production

DEV

OPS
Area 2: Extend operations
feedback to project

Area 4: Embed operations knowledge into project

Source: http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/ by Patrick Debois

DevOps Tools - Overview

IT Automation: Puppet
Shared Version Control: Perforce
Build and Deploy: Atlassian Bamboo
Process Management: Atlassian JIRA
6

IT Automation with Puppet

How Puppet works

Define.

Simulate.

Enforce.

Report.

Modules

Define Composable Deployments with Puppet

1
Database

Web Server

Via Puppet Dashboard

Application Server

Security

Custom External Source

(CMDB, LDAP, etc.)

Web Servers

Database Servers

Define your
resources in
modules.

Assign resource
relationships
automatically.

Application Servers

Reuseable,
composable
configurations.

Puppet Resource Abstraction Layer (RAL)

user { 'dave':
ensure
=>
uid
=>
gid
=>
shell
=>
home
=>
managehome =>
}

present,
'507',
'admin',
'/bin/zsh',
'/home/dave',
true,

10

Puppet RAL Example: class ssh

class ssh {
package { ssh:
ensure => latest,
}
file { /etc/ssh/sshd_config:
source => puppet:///modules/ssh/sshd_config,
}
service { sshd:
ensure => running,
}
Package[ssh]->File[/etc/ssh/sshd_config]~>Service[sshd]
Package[ssh]~>Service[sshd]
}
11

Puppet Modules Deployments as Code

module_name

manifests

lib

files

spec

templates

tests

Modulefile

12

Puppet Modulefile for puppetlabs-apache

name 'puppetlabs-apache'
version '0.6.0'
source 'git://github.com/puppetlabs/puppetlabs-apache.git'
author 'puppetlabs'
license 'Apache 2.0'
summary 'Puppet module for Apache'
description 'Module for Apache configuration'
project_page 'https://github.com/puppetlabs/puppetlabsapache'
## Add dependencies, if any:
dependency 'puppetlabs/firewall', '>= 0.0.4'
dependency 'puppetlabs/stdlib', '>= 2.2.1'
13

http://forge.puppetlabs.com

14

Puppet Module Subcommand

puppet module install puppetlabs-apache --version 0.6.0

apache

stdlib

firewall

15

Sets of Modules
puppet module install puppetlabs-apache --version 0.6.0
puppet module install puppetlabs-ntp --version 0.2.0

apache

stdlib

firewall

ntp

16

Sets of Modules Puppet Environments

puppet module install puppetlabs-apache --version 0.6.0
puppet module install puppetlabs-ntp --version 0.2.0

apache

stdlib

firewall

ntp

Puppet Environment

Multiple Environments on a single Puppet Master

17

Versioning Puppet Modules/Environments

Option A: Version each Module you need to change locally

apache

stdlib

firewall

ntp

Puppet Environment

A Git Repository for each Puppet Module you change

18

Versioning Puppet Modules/Environments

Option B: Version the Environment Folder

apache

stdlib

firewall

ntp

Puppet Environment

A single Git Repository for the Puppet Environment

19

Versioning Puppet Modules/Environments

Option C: Version the Environment Folder and all Modules

apache

stdlib

firewall

ntp

Puppet Environment

A single Git Repository for the Puppet Environment that uses Git
Submodules to include the Git Repositories for the Modules
20

Managing Puppet Environments with Perforce Git Fusion

environment

apache

main
forge
stdlib
apache
stdlib
firewall

firewall

21

Managing Puppet Environments with Perforce Git Fusion

environment

apache

main
forge
stdlib
apache
stdlib
firewall

firewall
Puppet Environment

22

Mainline Model for Puppet Modules

forge

hotfix_1

release_12.3

release_13.1

mainline

feature_1

feature_2

feature_3

feature_4

23

Perforce Git Fusion - Benefits

Modularity Puppet Environment Repository is
assembled from multiple Module Repositories
Easy Workflow - NO Git Submodules
Central Change History in Perforce
Fine grained Permissions

24

Puppet Architecture
Puppet Module Forge

Process Management

Version Management

Build and Deploy

Modules/Environments

Puppet Console

Puppet Master
Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

25

Puppet Data Flow

Node

1 Facts

SSL secure
Encryprion
on all data
transport

2 Catalog

Report 3
Puppet
Master

4
Report

Report
Collector

26

Scaling DevOps
Agile Flow of Change for Puppet Modules

27

What means Scaling DevOps?

More Teams/Services adopt DevOps Practices
Number of Environments and Modules grows

Fine grained Security becomes more important

Central Change history is required

Processes are standardized

Environments are standardized
Puppetized Deployments are used across all Stages
28

Agile Flow of Change for Puppet Modules

hotfix_1

release_12.3

release_13.1

mainline

apache

rails_webapp

feature_1

feature_2

zend_webapp

java_app

29

Agile Flow of Change for Puppet Modules

rel_12.3

rel_13.1

main

apache

apache_only

java_app

rails_webapp

feature_1

zend_webapp

feature_2

30

Perforce Streams Modularity

main
rails
webapp

apache

zend
webapp

zend_webapp

rails_webapp
rails
webapp

zend
webapp

All trademarks and logos are property of their respective owners

31

Build Automation - The Puppet Environment Lifecycle

Development

Quality Assurance

Staging

Production

Feature Stream

Service Stream

Mainline Stream

Release Stream/Tag

32

Summary
Puppet Module Forge

Process Management

Version Management

Build and Deploy

Modules/Environments

Puppet Console

Puppet Master
Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

Puppet Agent Facter

NODE

33

Thank you.
perforce@celix.at

34