Dynamic Jenkins Agents with Kubernetes

5 min readJul 29, 2021

Introduction

Jenkins is a CI/CD tool with a long history and keeps evolving itself. It’s Master/Agent architecture is great for scalability to do distributed builds. There are many ways to provision Jenkins Agent, from using bare metal machines, Virtual Machines, dynamic EC2 instances, containers from Docker, or Kubernetes clusters.

The integration between Jenkins and Kubernetes cluster is great. With these benefits, I’ve fully migrated CI/CD pipeline from host(VM) based Agent to Pod based Agent.

Dynamic Jenkins agent from Kubernetes, light weighted, provisioned on-demand within a few seconds
Fresh and reproducible Jenkins agent environment for every build
Resource/cost saving brought by Kubernetes

Pre-requisites:

Fully working Kubernetes cluster

Jenkins instance

Setup:

I am assuming that we already have Kubernetes, if not, this guide will help you to setup a local Kubernetes cluster Minikube.

As we already know Jenkins have wide range of plugins to integrate with the other tools which are available in the market. so in our requirement we need Kubernetes plugin to be installed to connect with the Kubernetes cluster.

To install the plugin please click on Manage Jenkins → Manage Plugins and click the ‘Available’ tab to search for ‘Kubernetes’. Select the check box next to it, and click on ‘Download now and install after restart’.

Once Plugin is installed, restart your Jenkins to load the plugin. you can simply restart the Jenkins after keeping `restart` after your URL.

http://<Jenkins Server IP>:8080/restart

To authenticate with Kubernetes cluster we can use multiple authentications which are available in Kubernetes, but in this case I’m simply using the config file which is generated by Kubernetes. So to add the secret, please navigate to Manage Jenkins → Manage Credentials → click on Global → Add Credentials → Select kind as secret file → Give all the necessary details (uploading config file and names, etc) and click on ‘ok’ button.

Now we are good to integrate the Jenkins with Kubernetes. To do this, we should navigate to Manage Jenkins → Manage Nodes & Clouds → Configure Clouds → add a new cloud and Select Kubernetes → Update the values as necessary.

Values:

Kubernetes URL: Since we are just passing the config file and the file contains all the cluster related details we no need to pass this argument. (it’s mandatory if we are going with the certificate base or token base authentication).
Kubernetes server certificate key: This option is only needed when we are trying to Authenticate the cluster with the certificates. In our use case it’s optional.
Kubernetes Namespace: Namespace where your build pods will get created. So entirely our choice. In this case, I’m keeping it blank, so that it will create pods in default name space.
Credentials: Here I am selecting the credentials which I have created.
Jenkins URL: URL for your Jenkins master and which is mandatory too. (http://<Jenkins Server IP>:8080)
Jenkins tunnel: This is to connect the Jenkins slave back to the Jenkins master, so we need to provide the Jenkins master IP along with the TCP Tunnel Port. IN our case I ma using 50000 port. So my URL is like Jenkins Server IP>:50000
Max connections to Kubernetes API : Number of simultaneous connections Jenkins is allowed to send to Kubernetes.

Creating Pipeline:

Create a new Pipeline Job by clicking on ‘New Item’ on the Jenkins home page and selecting the Pipeline type.

Once you click ‘ok’ we get the configure page. On the configure page scroll down to last and for ‘Definition’ choose ‘Pipeline script from SCM’, which will clone your repo defined into the workspace when your build starts. This is the only setup your job will need as everything else is in your Jenkinsfile.

Declarative Pipeline plugin integrates very nicely with the Declarative Pipeline syntax, allowing you to define your agents pod spec from your project’s source control, as shown below.

demo.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: sarva-test
spec:
  containers:
   - name: mavencontainer
     image: maven:3.5.4-jdk-11-slim
     command: ["sleep", "43200"]
   - name: dockercontainer
     image: docker:20
     command: ["sleep", "43200"]
     volumeMounts:
       - name: docker
         mountPath: /var/run/docker.sock
  volumes:
    - name: docker
      hostPath:
        path: /var/run/docker.sock

Jenkinsfile:

pipeline {
  agent {
    kubernetes {
      yamlFile 'demo.yaml'
      idleMinutes 1
    }
  }
  stages {
    stage('Build maven code') {
      steps {
      container('mavencontainer') {  
          sh "mvn clean install"
       }    
      }
    }
    stage('Build docker image') {
      steps {
        container('dockercontainer') {
          withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'pass', usernameVariable: 'user')]) {
          sh '''
          docker login -u ${user} -p ${pass}
          docker build -t jsarvabhowma/demoapp:v1 .
          docker push jsarvabhowma/demoapp:v1
          '''
          }
        }
      }
    }
  }
}

Work Behavior:

This is what will happen when you click ‘Build Now’ on our pipeline, shown in the Jenkinsfile snap below.

Jenkins clones your project into the workspace, reads the Jenkinsfile to determine the agent type.
A Kubernetes Pod with the spec defined in demo.yaml is created in the namespace defined in the general settings.
Project is build using mvn clean install on our project which will create a jar file inside the target directory.
A Docker Image is produced by copying the jar file from the target directory.

Results:

The first part of the build logs show that first the Jenkinsfile is loaded then any configured Jenkins Shared Libraries. Finally, your Kubernetes Pod spec gets sent to the Kubernetes API Server to get created.

After all the source code is loaded and the agent is created, your pipeline will begin executing any stages you defined, building our Java app with Maven and packaging it with Docker.