Container Days ATX

Advanced Docker

Created by Paul Czarkowski / @pczarkowski

Who Am I ?


Cloud Engineer - BlueBox Cloud

Blue Box Cloud

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery

Optimizing Dockerfiles



Blog Post

Optimizing Dockerfiles



  • Docker images are “supposed” to be small and fast.
  • Software is complicated, languages like Python/Ruby can result in large images.
  • Dockerfile != $ curl | sudo bash

Optimizing Dockerfiles


FROM ubuntu:latest

ADD . /home/docker

ADD https://dist.torproject.org/torbrowser/4.0.4/tor-browser-linux64-4.0.4_en-US.tar.xz /home/docker/tor.tar.xz

RUN apt-get update && true

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y firefox

RUN localedef -v -c -i en_US -f UTF-8 en_US.UTF-8 || :

RUN useradd -m -d /home/docker docker

RUN cd /home/docker && tar xJf /home/docker/tor.tar.xz

USER docker

CMD ["/home/docker/tor-browser_en-US/start-tor-browser"]

Optimizing Dockerfiles

Order Matters



Optimizing your Dockerfile for build cache hits.

  • The more cache hits the faster the build
  • First command to change invalidates all cache
  • Smart ordering can save minutes

Optimizing Dockerfiles

Order Matters



Sort your commands by

  1. Sharable with other images
  2. Frequency of change
  3. Time to run

Optimizing Dockerfiles

Know your commands



  • ADD always invalidates cache
  • WORKDIR, CMD are cheap

Optimizing Dockerfiles

Choose your base images wisely



  • Remember you're using a layered FS
  • ubuntu:trusty + busybox > 2 x Ubuntu:trusty

Optimizing Dockerfiles

Layers are your friend



  • Each command is a layer
  • Join RUN commands together with && \
  • A shared layer is free for subsequent containers.
  • Create your own common base image from when lots of shared layers.

Optimizing Dockerfiles

Cheat!



  • Add new changes to bottom of Dockerfile
  • Once locked in, then move them up.

Optimizing Dockerfiles

Volume Containers



Use the image of the application that is consuming it as the base for the volume container.



  • It's free because it's shared.
  • Volume container now contains the Data and the Application.

Optimizing Dockerfiles


FROM ubuntu:latest

ADD . /home/docker

ADD https://dist.torproject.org/torbrowser/4.0.4/tor-browser-linux64-4.0.4_en-US.tar.xz /home/docker/tor.tar.xz

RUN apt-get update && true

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get install -y firefox

RUN localedef -v -c -i en_US -f UTF-8 en_US.UTF-8 || :

RUN useradd -m -d /home/docker docker

RUN cd /home/docker && tar xJf /home/docker/tor.tar.xz

USER docker

CMD ["/home/docker/tor-browser_en-US/start-tor-browser"]

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Docker vs Config Management



"I don't need Config Management now that I have Docker."

Docker vs Config Management



Docker vs Config Management



You probably shouldn't be trusting docker [yet] for managing long term persistent data.

  • Volume Containers
  • Volume mounts from Host

Docker vs Config Management



Persistent data stores and other infrastructure services have to be managed by something.

  • Chef recipes to spin up databases, Key Value stores, etc.
  • Ansible or Cloud Formations for RDS, ELB, etc.

Docker vs Config Management

Chef and Docker



[the tools, not the companies]
have a great history of working together.

https://supermarket.chef.io/cookbooks/docker 

include_recipe "docker::default"

Docker vs Config Management


docker_image 'registry' do
  action [:pull]
end

docker_container 'registry' do
  detach  true
  port    '5000:5000'
  action [:run]
end

node['my_app']['images'].each do |image|
  docker_image image do
    action [:pull]
  end
  docker_image image do
    repository "#{node['my_app']['registry']}/#{image}"
    registry   node['my_app']['registry']
    action [:tag, :push]
  end
end

Docker vs Config Management


git "#{Chef::Config[:file_cache_path]}/docker-testcontainerd" do
  repository 'git@github.com:bflad/docker-testcontainerd.git'
  notifies :build, 'docker_image[tduffield/testcontainerd]', :immediately
end

docker_image 'tduffield/testcontainerd' do
  action :pull_if_missing
end

Docker vs Config Management



Almost anything that can be done with docker can be done via the docker cookbook.

Puppet and Ansible both have a similar story.

Docker vs Config Management

Config Management in the container ?



You probably shouldn't...

But if it makes sense in your use case ... go for it.

Consider Chef for Containers if you want to do this, it will help deal with managing services etc.

Run Chef from a volume mount so that it doesn't actually live in your container.

Docker vs Config Management

Config Management to build images ?



If you're already using a tool like Packer to create images with Config Management, you can probably switch it to build Docker images very easily.

Docker vs Config Management

ChefDK in a container?



If you're like me and run an OS that's not well supported by ChefDK ... run it in a container. 


$ docker run -d -v ~/chef/cookbook:/cookbook --name chefdk spheromak/docker-chefdk sleep 6000000
$ docker exec -ti chefdk bundle install
$ docker exec -ti chefdk rake test

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Logging and Metrics



This used to be hard. It's a lot less hard now.

Logging and Metrics



Try your very best not to ever log to a file in the container. Otherwise you will probably face logfile management issues.

Logging and Metrics



Logspout by @progrium

  • Runs in a container, watches the docker log stream
  • all stdout/sderr is seen by logspout
  • can be set to forward to a syslog server. 

$ docker run -v=/var/run/docker.sock:/tmp/docker.sock \
		progrium/logspout \
		syslog://logs.papertrailapp.com:55555

Logging and Metrics



But my app doesn't log to stdout/stderr.

It does now! 


# /etc/nginx/nginx.conf
worker_processes 1;
daemon off;
user app app;
pid /app/nginx.pid;
error_log /dev/stderr;
access_log /dev/stdout;
...

Logging and Metrics



But seriously I can't configure it to log to /dev/stdout

  • symlink /dev/stdout to the known location of logfile.
  • run syslog in container, push directly to your syslog server.
  • logspout will hopefully soon also include a syslog server.
  • run a volume container and mount in path for logs
  • log to volume mount from host.

Logging and Metrics



Logging improvements coming soon!

Logging and Metrics



  • Docker is a wrapper for LXC.
  • LXC exposes metrics in /proc
  • Example - /sys/fs/cgroup/memory/lxc/[longid]/
  • Hard to track where all of these metrics live.

Logging and Metrics



CAdvisor to the rescue. 


sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

Logging and Metrics

CAdvisor



Cadvisor natively support Docker and most linux containers based on LXC

  • Web UI - Demo Later
  • Rest API
  • InfluxDB and Prometheus outputs.
  • Probably needs to support Graphite.

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Multiple Processes



There are times when you need to run multiple processes in a container.

Ignore the nay-sayers ... this is OK.

Multiple Processes

Backgrounded in a start script




#!/bin/boot

mysqld_safe &
/usr/bin/hhvm --config /etc/hhvm/server.ini --user app -m server &
apache2 -DFOREGROUND

Multiple Processes

Backgrounded in a start script



Problems

  • Any process could die, container would still run.
  • Zombie Processes ( [defunc] )
  • Trying to resolve a solved problem ( init systems )
  • use exec any time calling "pid 1" app from script in docker.

Multiple Processes

Init system ... Runit



  • Simple lightweight init system
  • place executable script in /etc/services/[NAME]/run
  • restarts processes on exit/crash
  • `$ docker exec wordpress service apache2 restart`
  • handles zombie processes ( I think! )

Zombie Processess



Docker and the pid 1 zombie reaping problem

  • When process dies, turns into zombie process.
  • Parent process must explicitly wait for child process termination.
  • Is this really a big deal? Probably not...
  • However it can be avoided - Phusion - my_init
  • myinit when called with no arguments will start runit and let runit handle service mangement.

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Container Centric OSes



  • CoreOS
  • Atomic
  • Snappy ( Ubuntu Core )
  • RancherOS ( brand new! )

CoreOS



  • Free with pay for support & premium features
  • A fork of ChromeOS
  • FastPatch for updates
  • No package manager!
  • Clustering support.
  • Rocket

CoreOS

Clustering Support



  • etcd
  • fleet
  • flannel*

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

App Configuration

via ENV or service discovery



I have an application in a container ... but how do I actually configure it ?

App Configuration

During `docker build`



This is the simplest use case, if your configuration file is very static just use ADD in your `Dockerfile` to put it in the correct place.

App Configuration

via volume mount



You can also mount it in from your host ( useful if already using CM to write the config file on the host ). 


docker run -d -v /my/app/apache2/httpd.conf:/etc/apache2/httpd.conf myapp

It's OK to do this, but be aware of the potential security implications.

App Configuration

As part of a start script




#!/bin/bash
PORT=${PORT:-5000}
sed -i "s/xxxPORTxxx/$PORT/" /etc/apache2/httpd.conf
exec apache2 -DFOREGROUND

App Configuration

Using confd



  • lightweight templating engine written in GO.
  • Take key/value pairs from a number of supported storage engines.
  • Environment Variables, ETCD, Consul, etc.

App Configuration

Using confd




FROM ubuntu:trusty
RUN \
  curl -sSL -o /usr/local/bin/etcdctl \
  		https://s3-us-west-2.amazonaws.com/opdemand/etcdctl-v0.4.6 \
  && chmod +x /usr/local/bin/etcdctl \
  && curl -sSL -o /usr/local/bin/confd \
  		https://github.com/kelseyhightower/confd/releases/download/v0.7.1/confd-0.7.1-linux-amd64 \
  && chmod +x /usr/local/bin/confd

App Configuration

Using confd




#!/bin/bash

BACKEND=${BACKEND:-env}
until confd -onetime -backend=${BACKEND} -confdir=/etc/confd	; do
  echo "echo ==> ${APP_NAME}: waiting for confd to write initial templates..."
  sleep 1
done

exec apache2 -DFOREGROUND

App Configuration

Using confd




$ docker run -d -p 8080 \
  -e APACHE_PORT=8080 -e APACHE_ROOT=/app/web \
  apache /app/bin/boot

App Configuration

Using confd



Deeper dive into confd during demo / workshop.

Agenda



  • Optimizing Dockerfiles
  • Docker vs Config Management
  • Logging and Metrics
  • Running multiple processes in a container
  • Container Centric OSes
  • App Configuration via ENV or service discovery
  • Demo / Workshop

Demo / Workshop



Factorish - Example App

docker pull factorish/example

Demo / Workshop


$ docker run -d -e SERVICES_EXAMPLE_TEXT=father \
  -p 8080:8080 --name father factorish/example
$ curl localhost:8080 # or IP of boot2docker

$ docker run -d -e SERVICES_EXAMPLE_TEXT=mother \
  -p 8081:8080 --name mother factorish/example
$ curl localhost:8080 # or IP of boot2docker

$ docker kill father mother
$ docker rm father mother

Demo / Workshop


$ vagrant up
.....
$ vagrant ssh core-01
$ etcd ctl / --recursive

Got Questions?

THE END

BY Paul Czarkowski / @pczarkowski



For Container Days ATX 2015!