How to enable PSP with Traefik

If you are reading this, it is possibly because you already know what Traefik is and you want to use it without running it as root (main proces pid 1) or any other privileged user. If security is a big consern, enabling Kubernetes PodSecurityPolicy - PSP is considered to be one of the best-practices when it comes to safety mechanisms.

To do that, we were experimenting with building a container image for Traefik that uses libcap-progs and authbind configurations. If you are curious enough, feel free to read our source-code. Last but not least, this image not officially part of openSUSE (yet) but after some testing (yes sir, we test our container images) we hope to make there.

How to use it

First off, you need a working Kubernetes cluster. If you want to follow along with this guide, you should setup a cluster by yourself. This means either using openSUSE Kubic along with kubeadm or following the upstream Traefik instructions you can also use minikube on your machine, as it is the quickest way to get a local Kubernetes cluster setup for experimentation and development. In thend, we assume that you have kubectl binary installed in your system.

$ sudo zypper in minikube kubernetes-client
$ minikube start --vm-driver=kvm2

Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Downloading Minikube ISO
 181.48 MB / 181.48 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.13.2
Downloading kubelet v1.13.2
Finished Downloading kubeadm v1.13.2
Finished Downloading kubelet v1.13.2
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
Verifying kubelet health ...
Verifying apiserver health ...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

Everything looks great. Please enjoy minikube!

By now, your client and your cluster should already be configured:

$ kubectl get nodes

NAME       STATUS   ROLES    AGE    VERSION
minikube   Ready    master   103s   v1.13.2

Authorize Traefik to use your Kubernetes cluster

The new kubernetes versions are using RBAC (Role Based Access Control) that allows Kubernetes resources to communicate with its API under a controlled manner. There are two ways to set permissions to allow the Traefik resources to communicate with the k8s APIs:

• via RoleBinding (namespace specific)
• via ClusterRoleBinding (global for all namespaced)

For the shake of simplicity, we are going to use ClusterRoleBinding in order to grant permission at the cluster level and in all namespaces. The following ClusterRoleBinding allows any user or resource that is part of the ServiceAccount to use the traefik ingress controller.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:     ['policy']
    resources:     ['podsecuritypolicies']
    verbs:         ['use']
    resourceNames: ['traefik-ingress-controller']
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

$ kubectl apply -f rbac.yaml

serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

Notice that part of definition of the ClusterRole is to force it to use (verb) the podsecuritypolicy.

Enable PSP

We are going to enable a PodSecurityPolicy that disallow root user to run our Traefik container:

---
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: traefik-ingress-controller
spec:
  allowedCapabilities:
    - NET_BIND_SERVICE
  privileged: false
  allowPrivilegeEscalation: true
  # Allow core volume types.
  volumes:
    - 'configMap'
    - 'secret'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false
  seLinux:
    rule: 'RunAsAny'
  hostPorts:
    - max: 65535
      min: 1

$ kubectl apply -f podsecuritypolicy.yaml

podsecuritypolicy.extensions/traefik-ingress-controller created

You can verify that is loaded by typing:

$ kubectl get psp
NAME                         PRIV    CAPS               SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
traefik-ingress-controller   false   NET_BIND_SERVICE   RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,secret

Deploy our experimental Traefik image

I am going to deploy Traefik as a deployment kind using NodePort.

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: registry.opensuse.org/devel/kubic/containers/container/kubic/traefik:1.7
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
        securityContext:
          capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
          runAsUser: 38

As you can see our Traefik image is using authbind and setcap to enable a normal user (with 38 uid) to open ports lower than 1024.

$ kubectl apply -f deployment.yaml

deployment.extensions/traefik-ingress-controller created

To verify that is up and running, list your pods at kube-system namespace:

kubectl -n kube-system get pods | grep traefik

traefik-ingress-controller-87cbbbfb7-stlzm   1/1     Running   0          41s

In addition, you can also query for the deployments:

$ kubectl -n kube-system get deployments

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
coredns                      2/2     2            2           106m
traefik-ingress-controller   1/1     1            1           27m

To verify that Traefik is running as normal user (name should be traefik with UID 38):

traefikpod=$(kubectl -n kube-system get pods | grep traefik | awk '{ print $1 }')
kubectl -n kube-system exec -it $traefikpod -- whoami && id

traefik
uid=1000(tux) gid=100(users) groups=100(users),469(docker),472(libvirt),474(qemu),475(kvm),1003(osc)

So far we do not have a service to access this. It is just a Pod, which is part of the deployment.

$ kubectl -n kube-system expose deployment traefik-ingress-controller --target-port=80 --type=NodePort

service/traefik-ingress-controller exposed

You can verify this by quering for services under the kube-system namespace:

$ kubectl get svc -n kube-system

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
kube-dns                     ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP                 107m
traefik-ingress-controller   NodePort    10.105.27.208   <none>        80:31308/TCP,8080:30815/TCP   2s

We see that the traefik-ingress-controller service is becoming available on every node at port 31308 – the port number will be different in your cluster. So the external IP is the IP of any node of our cluster. You should now be able to access Traefik on port 80 of your Minikube cluster by requesting for port 31308:

$ curl $(minikube ip):31308

404 page not found

Note: We expect to see a 404 response here as we haven’t yet given Traefik any configuration.

The last step would be to create a Service and an Ingress that will expose the Traefik Web UI. From now on you can actually use the official Traefik documentation:

kubectl apply -f https://raw.githubusercontent.com/containous/traefik/master/examples/k8s/ui.yaml

Now lets setup an entry in our /etc/hosts file to route traefik-ui.minikube to our cluster.

In production you would want to set up real DNS entries. You can get the IP address of your minikube instance by running minikube ip:

echo "$(minikube ip) traefik-ui.minikube" | sudo tee -a /etc/hosts

We should now be able to visit traefik-ui.minikube:<NODEPORT> in the browser and view the Traefik web UI.

Now, you should be able to continue reading the official traefik documentation and do all the cool stuff but with better security.

More fun?

In case you are using a full-blown kubernetes cluster using Kubic (meaning: have more than one nodes available at your disposal), feel free to setup a LoadBalancer at your hypervisor in which your hosting your Kubic virtual-machines:

sudo zypper in nginx

cat /etc/nginx/nginx.conf

load_module '/usr/lib64/nginx/modules/ngx_stream_module.so';
events {
     worker_connections  1024;
}

stream {
     upstream stream_backend {
         # server <IP_ADDRESS_OF_KUBIC_NODE>:<TRAEFIK_NODEPORT>;
         server worker-0.kubic-init.suse.net:31380;
         server worker-1.kubic-init.suse.net:31380;
         server worker-2.kubic-init.suse.net:31380;
     }

   server {
     listen ultron.suse.de:80;
     proxy_pass stream_backend;
   }
}

And then start the load balancer: sudo systemctl start nginx.

This means that anyone that visits my machine (that is ultron.suse.de in this example) will be redirected to one of my kubernetes nodes at nodeport (31380).

Have fun