Posts on Patrik Cyvoct

Tweaking an EFK stack on Kubernetes

Mon, 14 May 2018 00:00:00 +0000

This is the continuation of my last post regarding EFK on Kubernetes. In this post we will mainly focus on configuring Fluentd/Fluent Bit but there will also be a Kibana tweak with the Logtrail plugin.

Configuring Fluentd

This part and the next one will have the same goal but one will focus on Fluentd and the other on Fluent Bit. Our goal is to create a configuration that will separate the logs of different namespaces and select which containers we want to log depending on their label.

Configure the DaemonSet

The first thing we need to do is change Fluentd’s DaemonSet. In fact, if we use the one provided by Fluentd, the configuration file is hardcoded into the image and it is not very simple to change it. So we will create a Kubernetes ConfigMap and mount it in the /fluentd/etc folder. If you have RBAC enabled, and you should, don’t forget to configure it for Fluentd:

# fluentd-rbac.yml
# If you have RBAC enabled
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system

Now regarding the DaemonSet:

# fluentd-daemonset.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd # if RBAC is enabled
      serviceAccountName: fluentd # if RBAC is enabled
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.1-debian-elasticsearch
        env:
        - name:  FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name:  FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: "http"
        - name: FLUENT_ELASTICSEARCH_USER # even if not used they are necessary
          value: "foo"
        - name: FLUENT_ELASTICSEARCH_PASSWORD # even if not used they are necessary
          value: "bar"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluentd-config
          mountPath: /fluentd/etc # path of fluentd config file
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentd-config
        configMap:
          name: fluentd-config # name of the config map we will create

Note that we are using Fluentd v1. Some configurations will not work on v0.12!

You may wonder why I added FLUENT_ELASTICSEARCH_PASSWORD and FLUENT_ELASTICSEARCH_USER. It is because the Docker image fluent/fluentd-kubernetes-daemonset uses sed on the configuration file if these environment variables are not set, and since the ConfigMap is read-only the container will fail to start. We could change the base image of the DaemonSet but adding these two lines is simpler and doesn’t hurt.

With the DaemonSet created we can now focus on our fluentd-config ConfigMap.

Creating the ConfigMap

Here is a basic Fluentd configuration for Kubernetes (You can learn more on configuring Fluentd in their documentation):

# fluentd-config-map.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    
        # this tells fluentd to not output its log on stdout
        @type null
    

    # here we read the logs from Docker's containers and parse them
    
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
            
    

    # we use kubernetes metadata plugin to add metadatas to the log
    
        @type kubernetes_metadata
    

    # we send the logs to Elasticsearch
    
        @type elasticsearch
        include_tag_key true
        host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
        port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
        scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
        ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
        user "#{ENV['FLUENT_ELASTICSEARCH_USER']}" # remove these lines if not needed
        password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}" # remove these lines if not needed
        reload_connections true
        logstash_prefix logstash
        logstash_format true
        
            flush_thread_count 8
            flush_interval 5s
            chunk_limit_size 2M
            queue_limit_length 32
            retry_max_interval 30
            retry_forever true

The Kubernetes metadata plugin is already installed in the Docker image we use.

This configuration does about the same as the one provided by Fluentd. Now if you want for instance to not send the kube-system containers’ logs, you can add these lines before the Elasticsearch output:

                                                                            
    @type null

Split the logs regarding to the namespaces

Let’s assume you want to separate your logs depending on the container’s namespace. For instance you could send the logs from the dev namespace to one Elasticsearch cluster and the logs from the production namespace to another one. In order to achieve it we will use the rewrite tag filter. After the metadata plugin, we could add:

# this add the namespace name at the begining of the tag

    @type rewrite_tag_filter
    
        key $['kubernetes']['namespace_name']
        pattern ^(.+)$
        tag $1.${tag}

And then we could have something like that for the output:

# match the dev logs

    @type elasticsearch
    include_tag_key true
    host "#{ENV['FLUENT_ELASTICSEARCH_HOST_DEV']}"
    port "#{ENV['FLUENT_ELASTICSEARCH_PORT_DEV']}"
    scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME_DEV'] || 'http'}"
    ...

# match the production logs

    @type elasticsearch
    include_tag_key true
    host "#{ENV['FLUENT_ELASTICSEARCH_HOST_PROD']}"
    port "#{ENV['FLUENT_ELASTICSEARCH_PORT_PROD']}"
    scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME_PROD'] || 'http'}"
    ...

It’s just an example, let your imagination make the better of it :) !

Select which containers you want to log

Now we want to select which containers we want to log or which not to log. It is possible with the grep filter (This will only work on Fluentd v1 since nested keys does not seem to work on v0.12).

The idea here is to add a label to the containers you want to log or to the ones you don’t want to log. There is two approaches: either we label all the containers we want to log; or the ones that we don’t want to log.

For instance if we add fluentd: "true" as a label for the containers we want to log we then need to add:


    @type grep
    
        key $.kubernetes.labels.fluentd
        pattern true

Or similarly, if we add fluentd: "false" as a label for the containers we don’t want to log we would add:


    @type grep
    
        key $.kubernetes.labels.fluentd
        pattern false

And that’s it for Fluentd configuration. Again if you want some more configuration options, check the documentation of Fluentd and of the plugins we used.

Configuring Fluent Bit

Unfortunately configuring Fluent Bit to work just like we just did for Fluentd is not (yet?) possible. One way to achieve it would be to connect Fluent Bit to a Fluentd aggregator but I will not cover it here. You can find some information about it on the fluent Github repo.

Let’s tweak Kibana a bit with the Logtrail plugin

Logtrail is a plugin for Kibana to view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface inspired by Papertrail.

First we need to install the plugin (Kibana 5.X & 6.X only). To install the plugin you’ll need the URL of a Logtrail release. You can check them here.

You must take the URL corresponding to your Kibana version.

Now, you can build the image with the Logtrail plugin like this (assuming you want Kibana 6.2.4):

FROM docker.elastic.co/kibana/kibana-oss:6.2.4
RUN kibana-plugin install https://github.com/sivasamyk/logtrail/releases/download/v0.1.27/logtrail-6.2.4-0.1.27.zip
WORKDIR /config
USER root
RUN mv /usr/share/kibana/plugins/logtrail/logtrail.json /config/logtrail.json && \
    ln -s /config/logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json
USER kibana

Or pull the image from my Dockerhub: sh4d1/kibana-logtrail

I only have the 6.2.4 tag.

Next step is to configure Logtrail and we will use a ConfigMap. Here is the ConfigMap and the Deployment for Kibana:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logtrail-config
data:
  logtrail.json: |
    {
        "version" : 1,
        "index_patterns" : [
        {      
            "es": {
                "default_index": "logstash-*"
            },
            "tail_interval_in_seconds": 10,
            "es_index_time_offset_in_seconds": 0,
            "display_timezone": "local",
            "display_timestamp_format": "MMM DD HH:mm:ss",
            "max_buckets": 500,
            "default_time_range_in_days" : 0,
            "max_hosts": 100,
            "max_events_to_keep_in_viewer": 5000,
            "fields" : {
                "mapping" : {
                    "timestamp" : "@timestamp",
                    "hostname" : "kubernetes.host",
                    "program": "kubernetes.pod_name",
                    "message": "log"
                },
                "message_format": "{{{log}}}"
            },
            "color_mapping" : {
            }
        }]
    }     
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  labels:
    component: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: sh4d1/kibana-logtrail:6.2.4 # or your image
        volumeMounts:
          - name: logtrail-config
            mountPath: /config
        env:
        - name: CLUSTER_NAME
          value: myesdb # the name of your ES cluster
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http
      volumes:
        - name: logtrail-config
          configMap:
            name: logtrail-config

So let’s take look at Logtrail’s configuration. The first point is the default-index; it must be set to the index used by Elasticsearch. Then the important part is the fields section. It will display like:

timestamp hostname program:message

The message is defined in message_format. We could put something like {{{docker.container_id}}}: {{{log}}}. For further configuration you can check the repository of sivasamyk.

If you have any questions feel free to send me a email or contact me on the Docker Slack community @Sh4d1

How to deploy an EFK stack to Kubernetes

Wed, 18 Apr 2018 00:00:00 +0000

EFK stack? Like ELK?

You may have heard of ELK (Elasticsearch, Logstash, Kibana). EFK is the same stack where we replace Logstash by Fluentd. So why replace Logstash with Fluentd?

First of all, Fluentd is now hosted by the Cloud Native Computing Foundation, the same which hosts Kubernetes. Also, I find it very easy to configure, there is a lot of plugins and its memory footprint is very low.

We could also replace Fluentd by Fluent Bit, which is a lighter log forwarder but has fewer functionalities than the first one.

How does it work

A picture is worth a thousand words, so here is a simple schema.

Basically, each Fluentd container reads the /var/lib/docker to get the logs of each container on the node and send them to Elasticsearch. Finally, when we access Kibana, it requests the logs from Elasticsearch.

What for?

You may ask yourselves why bother setting up such a stack? I can just run docker logs ..., and there I have my logs.

Well, yes, you could. However, if you have several replicas you’ll have to go through each one of the containers to find what you are looking for. Here you can go through all the logs at once. Also, with Kibana you can create dashboards and very nice visualizations.

Take a look:

From elastic.co

Another interesting point is the manipulation of the logs. With Fluentd you can filter, modify and backup your logs very easily. But we’ll look into that in another post.

Enough with the introduction, let’s build this stack!

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

In the E*K stack, we use Elasticsearch to store and search the logs forwarded by Fluentd. To deploy it into our Kubernetes cluster, we can use the GitHub repository of pires: pires/kubernetes-elasticsearch-cluster

You can modify the values in es-master.yaml, es-client.yaml and in es-data.yaml, for changing the number of replicas, the names, etc.

By default, ES_JAVA_OPTS is set to -Xms256m -Xmx256m. This is a very low value but many users, i.e. minikube users, were having issues with pods getting killed because hosts were out of memory. One can change this in the deployment descriptors available in this repository. README.md

It is also recommended to change the storage from emptyDir to the storage of your choosing to get persistent storage of the data.

Once you have setup your files you can run:

$ kubectl -n logging create -f es-discovery-svc.yaml
$ kubectl -n logging create -f es-svc.yaml
$ kubectl -n logging create -f es-master.yaml
$ kubectl -n logging rollout status -f es-master.yaml
$ kubectl -n logging create -f es-client.yaml
$ kubectl -n logging rollout status -f es-client.yaml
$ kubectl -n logging create -f es-data.yaml
$ kubectl -n logging rollout status -f es-data.yaml

Note: I will run the EFK stack in the logging namespace, use kubectl create ns logging to create it.

Check that your Elasticsearch is up and running by following pires instructions.

If everything is okay, we can now set up Fluentd (or Fluent Bit)!

Fluentd

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. https://www.fluentd.org/architecture

Fluentd can be run directly on the host, or in a Docker container. Here we will use a DaemonSet to ensure that Fluentd is running on every node. The code is from fluent/fluentd-kubernetes-daemonset.

If you have RBAC enabled on your cluster (and I hope you have), check the ClusterRole, ClusterRoleBinding and ServiceAccount of fluentd-daemonset-elasticsearch-rbac.yaml.

Don’t forget to change the namespace to the one used when deploying Elasticsearch, it should be the same. Then we can apply the DaemonSet:


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    component: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      component: fluentd-logging
  template:
    metadata:
      labels:
        component: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd # if you have RBAC enabled
      serviceAccountName: fluentd # if you have RBAC enabled
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch" # the name of the previous es-svc.yml 
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200" # port of the previous es-svc.yml
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Fluent Bit

We will use the code of fluent/fluent-bit-kubernetes-logging. As for Fluentd, if you use RBAC, create the ClusterRole, ClusterRoleBinding and ServiceAccount with:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

We also need to create a ConfigMap for the configuration. You can either modify this one and apply it or use the default one:

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml

Once the ConfigMap is applied, here is the code for the DaemonSet:


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    component: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      component: fluent-bit-logging
  template:
    metadata:
      labels:
        component: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:0.12.17
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch" # the name of the previous es-svc.yml 
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200" # the port of the previous es-svc.yml 
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config # name of the previously created ConfigMap
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

Kibana

Once you have some logs into Elasticsearch, we can add a tool for exploring and analyze them like Kibana.

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack.

So we will deploy Kibana as a deployment. We’ll use the 6.2 OSS version because it does not have X-Pack enabled. (If you want X-Pack, you can, of course, adjust the image name).


apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: logging
  name: kibana
  labels:
    component: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana-oss:6.2.2
        env:
        - name: CLUSTER_NAME
          value: myesdb # name of the Elasticsearch cluster defined in the first part
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http

When the pod is up and running, we can make it accessible via a Service and an Ingress:


apiVersion: v1
kind: Service
metadata:
  namespace: logging
  name: kibana
  labels:
    component: kibana
spec:
  selector:
    component: kibana
  ports:
  - name: http
    port: 5601
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    namespace: logging
    name: kibana
    annotations:
        kubernetes.io/ingress.class: traefik
        ingress.kubernetes.io/auth-type: "basic"
        ingress.kubernetes.io/auth-secret: kibana-basic-auth
spec:
    rules:
      - host: ""
        http:
            paths:
                - path: /
                  backend:
                      serviceName: kibana
                      servicePort: http

For protecting Kibana with a user/password, we will use Traefik as an Ingress controller. To create the secret:

$ htpasswd -c ./auth 
$ kubectl -n logging create secret generic kibana-basic-auth --from-file auth
$ rm auth

When you will access to Kibana for the first time, you need to configure your first index. If you did not change Fluentd’s default configuration, the index name/pattern is logstash-YYYY-MM-DD. You can use a wildcard to catch all index beginning with logstash- by using logstash-*.

And now you have a functional EFK stack!

In the next part, we will dive into the configuration of Fluentd and Kibana.

Thanks to @jmaitrehenry for his feedback on this post.

Using Drone and Traefik on Kubernetes to deploy a basic application

Mon, 18 Dec 2017 00:00:00 +0000

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers.

That seems awesome, a CD built with containers in mind. Drone is written in Go, supports a lot of plugins, and is very easy to configure.

If you don’t know Traefik, you can check out my previous posts.

We will deploy Drone on a Kubernetes cluster, here using Google Cloud Platform (You can get free credits for one year). You can choose the provider you prefer, but the instructions may change.

Creation of the cluster

So first thing first, we need a running cluster. With Google Kubernetes Engine, it can be done with a simple command. I assume you have gcloud installed and a working Google Cloud account, if not check the doc here. In order to create the cluster we just have to type:

$ gcloud container clusters create  --cluster-version=1.8.3-gke.0
Creating cluster ...done.                                      
Created [https://container.googleapis.com/v1/projects//zones//clusters/].
kubeconfig entry generated for .
NAME  LOCATION        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
    1.8.3-gke.0       n1-standard-1  1.8.3-gke.0   3          RUNNING

We will be using Kubernetes 1.8+ so be sure to have the correct kubectl version.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3+f0efb3cb88375", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"not a git tree", BuildDate:"2017-11-09T12:38:03Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-gke.0", GitCommit:"86d3ac5eaf57223302c95e7d9fc1aeff55fb0c15", GitTreeState:"clean", BuildDate:"2017-11-08T21:42:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Using Helm to install Traefik

To use Traefik with Let’s Encrypt, the easiest way to install it is via Helm: the Kubernetes Package Manager. You can install it from here.

Once Helm is installed, you can run helm init. It will install Tiller (the cluster-side service) on your cluster. If you have several clusters, check that kubectl config current-context returns the wanted cluster, if not you can use the --kube-context option from helm to install it on a different one.

Let’s create our Traefik chart configuration. Here is my traefik-helm.yml

imageTag: 1.5                                    
ssl:
    enabled: true
    enforced: true
acme:
    enabled: true
    email: ""
    staging: false
    persistence.enabled: true
dashboard:
    enabled: true
    domain: ""

imageTag allows us to specify Traefik’s version, here the latest one. ssl.enabled, well, enables HTTPS while ssl.enforced enables the HTTP to HTTPS redirection. Next, the acme section is about Let’s Encrypt certificates generation, and the last section is whether or not you would like to set up Traefik’s dashboard. More configuration options can be found here, such as Volume types for persistent storage or connection to Prometheus. Also, I did not specify it, but by default, the replicas count will be one, which is the best option when using Let’s Encrypt for the moment.

We are now ready to run:

$ helm install --namespace kube-system --name traefik --values traefik-helm.yml stable/traefik
Error: release traefik failed: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system": Unknown user "system:serviceaccount:kube-system:default"

So that did not work as expected. We need to add a ClusterRoleBinding to add the Role cluster-admin to the ServiceAccount kube-system:default

$ kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
clusterrolebinding "add-on-cluster-admin" created

Now if we retry:

$helm install --namespace kube-system --name traefik --values traefik-helm.yml stable/traefik                                                  
NAME:   traefik                                                                                                                                 
LAST DEPLOYED: Sat Dec 16 11:05:39 2017                                                                                                         
NAMESPACE: kube-system                                                                                                                          
STATUS: DEPLOYED                                                                                                                                
                                                                                                                                                
RESOURCES:                                                                                                                                      
==> v1/ConfigMap                                                                                                                                
NAME             DATA  AGE                                                                                                                      
traefik-traefik  1     1s                                                                                                                       
                                                                                                                                                
==> v1/PersistentVolumeClaim                                                                                                                    
NAME                  STATUS   VOLUME    CAPACITY  ACCESS MODES  STORAGECLASS  AGE                                                              
traefik-traefik-acme  Pending  standard  1s

==> v1/Service
NAME                       TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
traefik-traefik-dashboard  ClusterIP     10.43.254.78          80/TCP                      1s
traefik-traefik            LoadBalancer  10.43.249.218      80:30922/TCP,443:30670/TCP  1s

==> v1beta1/Deployment
NAME             DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
traefik-traefik  1        1        1           0          1s

==> v1beta1/Ingress
NAME                       HOSTS        ADDRESS  PORTS  AGE
traefik-traefik-dashboard    80       1s

==> v1/Pod(related)
NAME                              READY  STATUS   RESTARTS  AGE
traefik-traefik-6fc99c65d7-qm44j  0/1    Pending  0         1s

==> v1/Secret
NAME                          TYPE    DATA  AGE
traefik-traefik-default-cert  Opaque  2     1s

NOTES:

1. Get Traefik's load balancer IP/hostname:

     NOTE: It may take a few minutes for this to become available.

     You can watch the status by running:

         $ kubectl get svc traefik-traefik --namespace
kube-system -w

     Once 'EXTERNAL-IP' is no longer '':

         $ kubectl describe svc traefik-traefik --namespace kube-system | grep Ingress | awk '{print $3}'

2. Configure DNS records corresponding to Kubernetes ingress resources to point to the load balancer IP/hostname found in step 1

It worked! We can get the external IP pointing to Traefik by this command (you may need to wait a few seconds):

$ kubectl describe svc traefik-traefik --namespace kube-system | grep Ingress | awk '{print $3}'
35.198.124.40

Now we just need to edit our DNS records to match our dashboard’s URL to this IP. Let’s say you have the example.com domain: You will need one A record from this IP to dashboard.example.com, one A record to drone.example.com, and a last to app.example.com. Here is what I get when accessing the dashboard:

Testing Traefik with an example

Now that we have Traefik up and running, and before getting started with Drone, we are just going to deploy a simple application to check if Traefik works as intended. Here is my basic-deployment.yml:

apiVersion: apps/v1beta2                              
kind: Deployment
metadata:
    name: nginx
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 2
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
              - name: nginx
                image: stenote/nginx-hostname:latest
---
apiVersion: v1
kind: Service
metadata:
    name: nginx
    labels:
        apps: nginx
spec:
    selector:
      app: nginx
    ports:
      - protocol: TCP
        port: 80
        name: http

It will create two replicas of a basic NGINX server which prints the hostname. We apply it with:

$ kubectl apply -f basic-deployment.yml          
deployment "nginx" created
service "nginx" created

Now we need to create an Ingress to tell Traefik to use this service as the endpoint for the app.example.com URL. Here is my basic-ingress.yml:

apiVersion: extensions/v1beta1                        
kind: Ingress
metadata:
    name: drone
    annotations:
        kubernetes.io/ingress.class: traefik
spec:
    rules:
      - host: ""
        http:
            paths:
              - path: /
                backend: 
                    serviceName: nginx
                    servicePort: http

And we apply it.

$ kubectl apply -f basic-ingress.yml                  
ingress "drone" created

If we visit and refresh several times this URL, we can see that the hostname is changing. Traefik is working and is correctly redirecting to one of the two replicas created, perfect!

Deploying and configuring Drone

Traefik is working so let’s get to Drone. I will set up Drone with GitHub, but you can do it with GitLab, Gogs, Bitbucket,…Only the environment variables will change. You can find the instructions here.

Configuring a GitHub OAuth

You must go to GitHub > Settings > Developer settings > OAuth Apps and click on New OAuth App. You must complete the form like this (replacing https://drone.foo.com with your URL):

Keep this page open since we will soon need the client and secret key.

Deploying Drone

We will deploy Drone on the K8s cluster, I will group all the necessary Kubernetes objects in one YAML file, but of course you can split it in several files. Here is my drone-k8s.yml.

apiVersion: apps/v1beta2                               
kind: Deployment                                       
metadata:                                              
    name: drone                                        
spec:                                                  
    selector:                                          
        matchLabels:                                   
            app: drone                                 
    replicas: 1                                        
    template:                                          
        metadata:                                      
            labels:                                    
                app: drone                             
        spec:                                          
            containers:                                
              - name: drone                              
                image: drone/drone:0.8                   
                env:
                  - name: DRONE_OPEN                       
                    value: "false"                         
                  - name: DRONE_ADMIN                      
                    value: ""                         
                  - name: DRONE_HOST                       
                    value: ""         
                  - name: DRONE_GITHUB                     
                    value: "true"                          
                  - name: DRONE_GITHUB_CLIENT              
                    value: ""          
                  - name: DRONE_GITHUB_SECRET              
                    value: ""
                  - name: DRONE_SECRET
                    value: ""
                ports:
                - containerPort: 8000
                  name: drone-web-port
                - containerPort: 9000
                volumeMounts:
                - mountPath: /var/lib/drone
                  name: drone-db
            volumes:
            - name: drone-db
              hostPath:
                path: /var/lib/drone
---
apiVersion: v1 
kind: Service                                          
metadata:                                              
    name: drone                                        
    labels:                                            
        apps: drone                                    
spec:                                                  
    ports:                                             
    - name: http                                       
      port: 80                                         
      targetPort: drone-web-port                       
      protocol: TCP                                    
    - name: agent                                      
      port: 9000                                       
    selector:                                          
      app: drone                                       
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
    name: drone-agent
spec:
    selector:
        matchLabels:
            app: drone-agent
    replicas: 1
    template:
        metadata:
            labels:
                app: drone-agent
        spec:
            containers:
              - name: drone-agent
                image: drone/agent:0.8
                args: ["agent"]
                env:
                  - name: DRONE_SERVER
                    value: "drone:9000"
                  - name: DRONE_SECRET
                    value: ""
                volumeMounts:
                  - mountPath: /var/run/docker.sock
                    name: docker-socket
            volumes:
              - name: docker-socket
                hostPath:
                    path: /var/run/docker.sock
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: drone
    annotations:
        kubernetes.io/ingress.class: traefik
spec:
    rules:
      - host: ""
        http:
            paths:
              - path: /
                backend:
                    serviceName: drone
                    servicePort: http

We have one Deployment and one Service for Drone’s server and one Deployment for Drone’s agent. We don’t need any Service for the agent since it does not expose any port. Finally, we have an Ingress for Traefik.

Okay, so now let’s apply it!

$ kubectl apply -f drone-k8s.yml
deployment "drone" created
service "drone" created
deployment "drone-agent" created
ingress "drone" configured

And, ta-da, you can access Drone at the URL you chose!

Using Drone

Now that Drone is working, I will show you how to use it. Drone will build the Dockerfile of my chosen repo, upload it to Dockerhub, tagged with the commit’s hash, and update the deployment on Kubernetes with the new upload image.

First, let’s create the .drone.yml in the repo you want to build.

pipeline:                                             
    docker:
        image: plugins/docker
        repo: /
        tags: ${DRONE_COMMIT_SHA:0:8}
        username: 
        secrets: [ docker_password ] 
    deploy:
        image: quay.io/honestbee/drone-kubernetes
        kubernetes_server: ${KUBERNETES_SERVER}
        kubernetes_cert: ${KUBERNETES_CERT}
        kubernetes_token: ${KUBERNETES_TOKEN}
        deployment: 
        repo: /
        container: 
        tag:
            - ${DRONE_COMMIT_SHA:0:8}

This file uses two plugins: one to push to Dockerhub and one to update the deployment.

To make it work, we need to add the right secrets. You will need to install the Drone CLI. On Linux, you just have to enter:

$ curl -L https://github.com/drone/drone-cli/releases/download/v0.7.0/drone_linux_amd64.tar.gz | tar zx
$ sudo install -t /usr/local/bin drone

Next step is to get the right tokens: go to your newly created Drone and on the top right click on the menu icon and click on Token. Copy and paste the first two lines of the Example CLI Usage into your terminal:

export DRONE_SERVER=""
export DRONE_TOKEN=""

If it worked, you should get the right output when you type drone info.

Adding secrets to Drone

Before adding any secret to a repo, this one needs to be added through Drone’s UI. For the first plugin to work, we need to add the docker_password secret:

drone secret add -repository / -image plugins/docker -name docker_password -value ""

The next plugins uses three different secrets: kubernetes_server, kubernetes_cert, and kubernetes_token.

However, we need to create an RBAC in order the plugin to deploy.

apiVersion: v1                                        
kind: ServiceAccount
metadata:
    name: drone-deploy
    namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: drone-deploy
    namespace: default
rules:
  - apiGroups: ["extensions"]
    resources: ["deployments"]
    verbs: ["get","list","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: drone-deploy
    namespace: default
subjects:
  - kind: ServiceAccount
    name: drone-deploy
    namespace: default
roleRef:
    kind: Role
    name: drone-deploy
    apiGroup: rbac.authorization.k8s.io

And apply it : kubectl apply -f drone-rbac.yml If you are running K8s on GKE, you need to first add the Role cluster-admin to your gcloud user (see this issue).

$ kubectl create clusterrolebinding  --clusterrole=cluster-admin --user=your.google.cloud.email@example.org

Then we need to extract the ca.crt and the token from the service account.

$ kubectl -n  get secrets
NAME                       TYPE                                  DATA      AGE
default-token-ms8cl        kubernetes.io/service-account-token   3         3h
drone-deploy-token-4zv8j   kubernetes.io/service-account-token   3         10m 
$ kubectl -n  get secret/drone-deploy-token-4zv8j -o yaml | grep 'ca.crt:'
  ca.crt: 
$ kubectl describe secret drone-deploy-token-4zv8j | grep 'token:'
  token:

Once this is done, we can create our secrets.

To get the IP of your master node, you can use the kubectl cluster-info command. And we create the secret with drone secret add --image=quay.io/honestbee/drone-kubernetes -repository / -name kubernetes_server -value "https://"
To create the token secret, we run drone secret add --image=quay.io/honestbee/drone-kubernetes -repository / -name kubernetes_token -value
To create the cert token, we run drone secret add --image=quay.io/honestbee/drone-kubernetes -repository / -name kubernetes_cert -value

However, for this plugin to work, the deployment has to be already created. You can check this plugin, if you want Drone to run kubectl apply -f deploy.yml instead of just changing the image of the deployment.

Now, once you push some new code on GitHub, it will trigger Drone and start your build/deployment. Cheers!

Multi HTTPS sub domain with Traefik and Docker - Part 2

Sun, 08 Oct 2017 00:00:00 +0000

In this (small) second part, we’ll set up Traefik Web UI on its own sub domain with a basic HTTP authentication.

Basic dashboard

In the traefik.toml we add the following just before the [entryPoints] label:

[web]
# Port for the status page
address = ":8080"

We also need to map the port 8080 in the docker-compose.yml so you have to add this line in the ports section:

- 8080:8080

Now we can access Traefik Web UI at yourdomain.com:8080. It will look like this:

Let’s add a sub domain

Now we need to modify the docker-compose.yml in order to redirect to a sub domain. The only thing we are going to add is the Docker labels:

"traefik.enabled=true"
"traefik.backend=dashboard"
"traefik.frontend.rule=Host:dashboard.yourdomain.com"
"traefik.port=8080"

Moreover, we don’t need to map the port 8080 since we will not access the dashboard through this port.

So our file will look like this:

version: '2'
services:
    traefik:
    image: traefik
    command: --web --docker
    ports:
      - "80:80"
      - "443:443"
    restart: always
    labels:
      - "traefik.enabled=true"
      - "traefik.backend=dashboard"
      - "traefik.frontend.rule=Host:dashboard.yourdomain.com"
      - "traefik.port=8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock" 
      - "./traefik.toml:/traefik.toml"
      - "./acme.json:/acme.json"
    networks:
      - default

Now you can access your dashboard through the desired URL!

A basic HTTP authentication

If you need to protect your application with a user/password authentication, Traefik can do it for you. Again it’s configurable via the Docker labels of the service: traefik.frontend.auth.basic=user:passwordHash You can get the hash with the htpasswd command like this

$ htpasswd -nB user
New password:
Re-type new password:
user:$2y$05$NiscFUPxmLub5vW1gL6cF.4R1ElHKeMgBQKNPIY.1V.CW802nXhwG

When you add the hash in the docker-compose.yml you must escape the $ character with another $. In this case we will have : traefik.frontend.auth.basic=user:$$2y$$05$$NiscFUPxmLub5vW1gL6cF.4R1ElHKeMgBQKNPIY.1V.CW802nXhwG. Your application will then be protected by a user/password authentication.

You can now easily access your public and private Docker applications through different sub domains.

Multi HTTPS sub domain with Traefik and Docker - Part 1

Wed, 04 Oct 2017 00:00:00 +0000

Today I’ll show you how to make several Docker containers accessible via automatic HTTPS with Let’s Encrypt on different sub domains. I assume you are already familiar with Docker, if not I recommend you to read the following getting started guide, especially the first three parts.

Traefik

We will use Traefik. So first what is Traefik? According to its website:

Traefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, and a lot more) to manage its configuration automatically and dynamically.

In our case, we will principally use the reverse proxy feature, and the automatic configuration. That means the right redirection from the sub domain to the right Docker.

Prerequisites

In order to follow this guide, you will need a server and a domain name. You can of course test Traefik on your local computer, but this is intended to real sub domains.

So I will consider that you have already:

a running and accessible server
a domain name and some sub domain names all pointing to your server IP
a working installation of Docker

Let’s dive into it

For the first example I’ll just create a basic web server using nginx. We’ll create the necessary directories

$ mkdir -p server/{traefik,nginx}

In the traefik folder, we can now create the docker-compose.yml file:

version: '2'
services:
  traefik:
    image: traefik:raclette
    command: --docker
    ports:
      - "80:80"
      - "443:443"
    restart: always
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./traefik.toml:/traefik.toml"
      - "./acme.json:/acme.json"
    networks:
      - default

Basically what it does is:

Pull the raclette (1.3) Traefik docker image
Start the command with the docker argument, which enable the Docker backend
Map the port HTTP and HTTPS ports to the host
Mount the Docker sock (to communicate with the Docker daemon)
Mount the configuration file and the Let’s Encrypt file Now we can create the traefik.toml and the acme.toml

Let’s Encrypt

Traefik uses Let’s Encrypt to generate automated HTTPS certificate for free. For more technical information you can visit the Let’s Encrypt website.

It will generate a certificate for each service we will create, and manage the renewal. The acme.json is here to store these certificates. We could keep it inside the Docker since if it reboots it will regenerate the certificates, however keeping it outside our container is a bit more efficient. (It could also avoid us to reach Let’s Encrypt limits). The file needs to have only read-write permissions for the current user so let’s create it and set the right permissions:

$ touch acme.json && chmod 600 acme.json

traefik.toml

The Traefik config file is in the toml format. Here is a short example with HTTP redirection to HTTPS and Let’s Encrypt certificate:

# defaultEntryPoints must be at the top 
# because it should not be in any table below

defaultEntryPoints = ["http", "https"]

# Entrypoints, http and https
[entryPoints]

# http should be redirected to https
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"

# https is the default
[entryPoints.https]
address = ":443"

[entryPoints.https.tls]

# Enable ACME (Let's Encrypt): automatic SSL
[acme]
email = "your.email@domain.com"
storage = "/acme.json"
entryPoint = "https"
onDemand = false
OnHostRule = true

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "yourdomain.com"
watch = true
exposedbydefault = false

You have to change your email, and the domain name. In this case, I disabled the exposedbydefault so we’ll have to manually tell that a container have to be exposed.

Run it!

Now that we have all the configuration file:

$ ls # in server/traefik
acme.json  docker-compose.yml traefik.toml

we can execute docker-compose up -d. It will create the Docker container for Traefik and a new network traefik_default. If we curl the sub domain desired for nginx we get a 404 page not found which indicates that Traefik received the requests but did not found any backend to proxy the request. Let’s set up our nginx container next!

Our first service

We succeeded to set up Traefik, we now have to use it. In order to use the Docker backend, we will use the Docker labels. Here are a few labels:

traefik.backend=example gives the name example to the generated backend
traefik.port=80 which indicates which port Traefik will proxy the requests to
traefik.enable=true enable Traefik on the specified container
traefik.frontend.rule=Host:subdomain.example.com tells Traefik to route all the requests from this sub domain to this container

The full list can be found here.

Here is the docker-compose.yml for the simple nginx container:

version: '2'
services:
  nginx:
    image: nginx:stable
    labels:
      - "traefik.enabled=true"
      - "traefik.backend=nginx"
      - "traefik.frontend.rule=Host:subdomain.example.com"
      - "traefik.port=80"
    restart: always
    networks:
      - traefik_default
networks:
  traefik_default:
    external:
      name: traefik_default

We don’t need the expose the port 80 since Traefik redirects the traffic in the container. And we have to add another network to have the Traefik container and the nginx container in the same network, so they can communicate. Another way would be the add the label traefik.docker.network with the right network name.

Finally you just have to docker-compose up -d and voilà! If you now try to open subdomain.example.com you will be redirected to https://subdomain.example.com with a valid certificate and you can see the nginx page.

A practical example

I’ll show you a quick example with two Docker containers running Flask on test1.ptrk.io and test2.ptrk.io. We need to create a directory test and the following files:

docker-compose.yml

version: '2'

services:
    test1:
        build: .
        labels:
          - "traefik.enabled=true"
          - "traefik.backend=test1"
          - "traefik.frontend.rule=Host:test1.ptrk.io"
          - "traefik.port=5000"
        networks:
          - "traefik_default"
        restart: always
    test2:
        build: .
        labels:
          - "traefik.enabled=true"
          - "traefik.backend=test2"
          - "traefik.frontend.rule=Host:test2.ptrk.io"
          - "traefik.port=5000"
        networks:
          - "traefik_default"
        restart: always
networks:
    traefik_default:
        external:
            name: traefik_default

Dockerfile

FROM python:2.7
WORKDIR /app
COPY . /app
RUN pip install flask
ENTRYPOINT ["python"]
CMD ["start.py"]

start.py

from flask import Flask, request

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello " + request.host


if __name__ == "__main__":
    app.run(debug=False,host='0.0.0.0')

Now if I run docker-compose up -d I can check test1.ptrk.io and test2.ptrk.io to see that we are correctly redirected to the right container.

Conclusion

You can found out more on the Traefik docs, and in my next post I’ll explain how to access to the Traefik web UI on another sub domain, and with basic HTTP authentication.