Note: If this topic has peaked your interest, you can join me for a Webinar on August 15 where I’ll dive deep into Cloud Native Operations with Kubernetes and CI/CD Pipelines.

Lulz what? Cloud Native Operations ?!?!?!

Historically Operations practices have lagged behind development. During the 90s a number of lightweight software development practices evolved such as Scrum and Extreme Programming. During the early 2000’s it became pretty common to practice (or at least claim to) some form of Agile in software development.

It wasn’t until the last year of that decade that we started to see an uptick in Operations folks wanting to adopt Agile type methodologies and as the devops (and later SRE) movements took off we started to borrow heavily from Lean principals such as Kanban and Value Stream Mapping.

Cloud Computing has brought about another shift in software development, going from large monolithic applications to collections of microservices that work together, and even further into being event based via messages and streams which now falls under the umbrella of “cloud native”.

With the rise of Kubernetes and similar platforms as well as companies like Hashicorp and stalwarts of Agile Operations such as Google and Pivotal we’re starting to see that same shift in Operations as we start to talk about Platform as Product and turning engineering [operations] teams into product teams.

this means we all need to learn to get better at product engineering, kittens. and turning infra engineering teams into infra product teams. pic.twitter.com/t1Vj6RKbdd

— Charity Majors (@mipsytipsy) January 25, 2019

Kubernetes Controllers

There’s a lot more to be said about Cloud Native Operations and Platform as Product (the two go hand-in-hand) but for now I want to focus on a fundamental aspect of Kubernetes that will be a force multiplier for making the composable building blocks of Cloud Native Operations.

Most resources in Kubernetes are managed by a Controller. A Kubernetes Controller is to microservices what a Chef recipe is to a Monolith.

Each resource is controlled by its own control loop. This is a step forward from previous systems like Chef or Puppet which both have control loops but at the server level, not the resource.

A Controller is a fairly simple piece of code that creates a control loop over a single resource to ensure that resource is behaving correctly. These Control loops cam stack together to create complex functionality with simple interfaces.

The canonical example of this in action is in how we manage pods in Kubernetes. A Pod is [effectively] a running copy of your application that a specific worker node is asked to run. If that application crashes the kubelet running on that node will start it again.

However if that node crashes the Pod is not recovered as the control loop (via the kubelet process) responsible for the resource no longer exists. To make applications more resiliant Kubernetes has the ReplicaSet controller.

Kubernetes has a process running on the masters called a controller-manager that run the controllers for these more advanced resources. This is where the ReplicaSet controller runs, and it is responsible for ensuring that a set number of copies of your application are always running.

To do this the ReplicaSet controller requests that the provided number of Pods are created and then it routinely checks that the correct number of Pods are still running and will request more pods, or destroy existing pods to do so.

By requesting a ReplicaSet from Kubernetes you get a self-healing deployment of your application. You can further add lifecycle management to your workload by requesting a Deployment which is a controller that manages ReplicaSets.

These Controllers are great for managing Kubernetes resources, but are also fantastic for managing resources outside of Kubernetes. You can extend Kubernetes by writing a Controller that watches for events and annotations and performs extra work, or by writing a Custom Resource Definition.

Example - External DNS Controller

The external-dns controller is a perfect example of a watcher. You configure it with your DNS provider and it will watch resources such as Services and Ingresses. When one of those resources changes it will inspect them for annotations which will tell it if it needs to perform an action.

With the external-dns controller running in your cluster you can simply add the following annotation to a service and it will go out and create a matching DNS A record for that resource:

kubectl annotate service nginx \
    "external-dns.alpha.kubernetes.io/hostname=nginx.example.org."

You can change other characteristics such as the TTL value of the DNS record:

kubectl annotate service nginx \
    "external-dns.alpha.kubernetes.io/ttl=10"

Just like that you now have automatic DNS management for your applications and services in Kubernetes that reacts to any changes in your cluster to ensure your DNS is correct.

Example - Certificate Manager Operator

Like the external-dns controller the cert-manager will react to changes in resources, but also comes with a Custom Resource Definition that will allow you to request certificates as a resource in of themselves, not just a byproduct of an annotation.

cert-manager works with Lets Encrypt and other sources of Certificates to request valid signed TLS certificates. You can even use it in combination with external-dns like the following which will register web.example.com and retrieve a TLS certificate from Lets Encrypt and store that in a Secret.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: "true"
  name: example
spec:
  rules:
  - host: web.example.com
    http:
      paths:
      - backend:
          serviceName: example
          servicePort: 80
        path: /*
  tls:
  - hosts:
    - web.example.com
    secretName: example-tls

You can also request a certificate directly from the cert-manager CRD like so which like above will result in a certificate keypair being stored in a Kubernetes secret:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example-com
  namespace: default
spec:
  secretName: example-com-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: example.com
  dnsNames:
  - www.example.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - example.com
    - http01:
        ingress: my-ingress
      domains:
      - www.example.com

Conclusion

This was just a quick look at one of the ways that Kubernetes is helping enable a new wave of changes to how we operate software. This is a favorite topic of mine, so look forward to hearing more.