Using the great Testcontainers library on Kubernetes isn’t all that straight forward but running your build pipelines on a Kubernetes cluster has many benefits and can be a great solution for moving towards auto scaling build environments. If you are using the Jenkins build server the Kubernetes plugin provides an easy way to set up agents running on your cluster. So is there no way to use Testcontainers and run your build on a Kubernetes cluster? I found a pretty good solution that I want to share with you today.

The Problem

Testcontainers talks directly to the Docker daemon on the system using the Docker java client. Running on Kubernetes, we usually do not have access to a Docker daemon. A method to work around this, if you want to build a Docker image on Kubernetes for example, is to mount the Docker socket of the host machine into the Kubernetes pod. This works for building Docker images but running a container and connecting to it from within the Pod won’t work using this method. While launching the container will probably work, talking to it will be pretty much impossible. Kubernetes creates its own network inside a pod and the container launched directly on the host won’t have access to the pod network.

Additionally, allowing pods or containers to mount a host path is usually discouraged for security reasons as this can be used to break out of the isolation provided by Docker and Kubernetes. For this reason the ability to mount host path should probably be disabled on your cluster.

The solution - using ”Docker in Docker”

The solution I found is the following: Another way to get access to a Docker daemon on Kubernetes is by running a Docker Docker container. Yes, those do really exist. The act of running Docker itself inside a Docker container has the creative name “Docker in Docker” and is a great way to get around some of the restrictions that mounting the Docker socket itself comes with.

Kubernetes allows us to run many Docker/Linux containers inside a Pod. If we have a pod with a main container and a utility or side container, the latter is usually called a sidecar container. This is what I did to my Jenkins build agent pods; they have a main container providing the tools necessary for building my Java application and a sidecar container running “Docker in Docker” to provide a Docker socket for Testcontainers. Theres also a container running the jnlp agent software so that the Pod is able to register itself with the Jenkins master.

All containers inside a Kubernetes Pod share the same host, so using the Docker socket of the sidecar container is as easy as setting the environment variable DOCKER_HOST to tcp://localhost:2375.

The example

Using a Jenkinsfile and, in this case, Scripted Pipelines setting up a build agent capable of the above can be done using the following code:

testLabel = "test-${UUID.randomUUID().toString()}"
podTemplate(label: testLabel, cloud: 'yourcloud', serviceAccount: 'jenkins-sa', containers: [
    containerTemplate(name: 'jnlp',
            image: 'jenkins/jnlp-slave:3.29-1-alpine',
            command: '/usr/local/bin/jenkins-slave',
            workingDir: '/home/jenkins',
            ttyEnabled: true,
            resourceLimitCpu: '1',
            resourceLimitMemory: '500Mi',
            resourceRequestCpu: '100m',
            resourceRequestMemory: '500Mi',
            envVars: []
    ),
    containerTemplate(name: 'jdk',
            image: 'adoptopenjdk:8u212-b04-jdk-hotspot',
            workingDir: '/home/jenkins',
            ttyEnabled: true,
            resourceLimitCpu: '2',
            resourceLimitMemory: '2Gi',
            resourceRequestCpu: '200m',
            resourceRequestMemory: '2Gi',
            envVars: [
                    envVar(key: 'DOCKER_HOST', value: 'tcp://localhost:2375'),
                    envVar(key: 'JAVA_TOOL_OPTIONS', value: '-Xmx700m'),
            ]
    ),
    containerTemplate(name: 'docker',
            image: 'docker:18.09.7-dind',
            workingDir: '/home/jenkins',
            ttyEnabled: true,
            resourceLimitCpu: '200m',
            resourceLimitMemory: '512Mi',
            resourceRequestCpu: '200m',
            resourceRequestMemory: '512Mi',
            privileged: true,
            envVars: [
            ]
    ),
], volumes: [emptyDirVolume(mountPath: '/var/lib/jenkins', memory: false), emptyDirVolume(mountPath: '/var/lib/docker', memory: false)]) {
    node(testLabel) {
        checkout()
        stage('test') {
            container('jdk') {
                sh "./gradlew test --no-daemon --stacktrace"
            }
        }
    }

}

Configuring the Pod Template can of course also be done via the Jenkins UI or using the Jenkins Configuration as Code Plugin. Duplicating the boilerplate code above into multiple Jenkinsfiles can be very cumbersome so you should find a way to globally provide sane default build agent Pod Templates.

Drawbacks

As you can probably tell from the configuration above, running a “Docker in Docker” container requires the container to run in privileged mode. To avoid any harm to the rest of your environment make sure that your build cluster or namespace is probably isolated from the rest of your cluster using service accounts and other access control mechanism provided by Kubernetes.

Conclusion

If you, like I was, are searching for a way to use the Testcontainers library in your Jenkins Kubernetes Build Agent I hope this post is of use for you. For any remaining questions feel free to contact me in the comments, on twitter, or via email.