By Steven Eschinger | March 22, 2017

This post was updated on September 18th, 2017 for Kubernetes version 1.7.6 & Kops version 1.7.0

Introduction

In the previous lab, we went through how to deploy Jenkins and integrate it with your Kubernetes cluster.

And in this lab, we will setup an example continuous deployment pipeline with Jenkins, using the same Hugo site that we have used in previous labs.

The four main stages of the pipeline that we will create are:

Build: Build the Hugo site
Test: Test the Hugo site to confirm there are no broken links
Push: Create a new Docker image with the Hugo site and push it to your Docker Hub account
Deploy: Trigger a rolling update to the new Docker image in your Kubernetes cluster

The configuration of the pipeline will be defined in a Jenkinsfile, which will be stored in your GitHub repository for the Hugo site.

And after configuring the GitHub plugin in Jenkins, the pipeline will be automatically triggered at every commit.

Activities

Warning: Some of the AWS resources that will be created in the following lab are not eligible for the AWS Free Tier and therefore will cost you money. For example, running a three node cluster with the suggested instance size of t2.medium will cost you around $0.20 per hour based on current pricing.

Prerequisites

Review the Getting Started section in the introductory post of this blog series
Log into the Vagrant box or your prepared local host environment
Update and then load the required environment variables:

# Must change: Your domain name that is hosted in AWS Route 53
export DOMAIN_NAME="k8s.kumorilabs.com"

# Friendly name to use as an alias for your cluster
export CLUSTER_ALIAS="usa"

# Leave as-is: Full DNS name of you cluster
export CLUSTER_FULL_NAME="${CLUSTER_ALIAS}.${DOMAIN_NAME}"

# AWS availability zone where the cluster will be created
export CLUSTER_AWS_AZ="us-east-1a"

# Leave as-is: AWS Route 53 hosted zone ID for your domain
export DOMAIN_NAME_ZONE_ID=$(aws route53 list-hosted-zones \
       | jq -r '.HostedZones[] | select(.Name=="'${DOMAIN_NAME}'.") | .Id' \
       | sed 's/\/hostedzone\///')

Implementation

Deploy a new cluster

Create the S3 bucket in AWS, which will be used by Kops for cluster configuration storage:

aws s3api create-bucket --bucket ${CLUSTER_FULL_NAME}-state

Set the KOPS_STATE_STORE variable to the URL of the S3 bucket that was just created:

export KOPS_STATE_STORE="s3://${CLUSTER_FULL_NAME}-state"

Create the cluster with Kops:

kops create cluster \
     --name=${CLUSTER_FULL_NAME} \
     --zones=${CLUSTER_AWS_AZ} \
     --master-size="t2.medium" \
     --node-size="t2.medium" \
     --node-count="2" \
     --dns-zone=${DOMAIN_NAME} \
     --ssh-public-key="~/.ssh/id_rsa.pub" \
     --kubernetes-version="1.7.6" --yes

It will take approximately 5 minutes for the cluster to be ready. To check if the cluster is ready:

kubectl get nodes

NAME                            STATUS    AGE       VERSION
ip-172-20-48-9.ec2.internal     Ready     4m        v1.7.6
ip-172-20-55-48.ec2.internal    Ready     2m        v1.7.6
ip-172-20-58-241.ec2.internal   Ready     3m        v1.7.6

Create the Hugo site

In this lab, we will create a Hugo site with the same theme (Material Docs by Digitalcraftsman) that we used in Labs #3 & #4.

From the root of the repository, execute the following, which will:

Create a new blank Hugo site in the hugo-app-jenkins/ folder
Clone the Material Docs Hugo theme to the themes folder (hugo-app-jenkins/themes)
Copy the example content from the Material Docs theme to the root of the Hugo site
Remove the Git folder from the Material Docs theme
Change the value of the baseurl in the Hugo config file to ensure the site will work with any domain

hugo new site hugo-app-jenkins/

git clone https://github.com/digitalcraftsman/hugo-material-docs.git \
    hugo-app-jenkins/themes/hugo-material-docs

cp -rf hugo-app-jenkins/themes/hugo-material-docs/exampleSite/* hugo-app-jenkins/
rm -rf hugo-app-jenkins/themes/hugo-material-docs/.git/
sed -i -e 's|baseurl =.*|baseurl = "/"|g' hugo-app-jenkins/config.toml

Create the GitHub repository for the Hugo site

Configure your global Git settings with your username, email and set the password cache to 60 minutes:

# Set your GitHub username and email
export GITHUB_USERNAME="smesch"
export GITHUB_EMAIL="steven@kumorilabs.com"

git config --global user.name "${GITHUB_USERNAME}"
git config --global user.email "${GITHUB_EMAIL}"
git config --global credential.helper cache
git config --global credential.helper 'cache --timeout=3600'

Create a new GitHub repository called hugo-app-jenkins in your GitHub account (you will be prompted for your GitHub password):

curl -u "${GITHUB_USERNAME}" https://api.github.com/user/repos \
     -d '{"name":"hugo-app-jenkins"}'

Create the README.md file and then upload the contents of the Hugo site (hugo-app-jenkins/) to the GitHub repository you just created (you will be prompted for your GitHub credentials):

echo "# hugo-app-jenkins" > hugo-app-jenkins/README.md
git -C hugo-app-jenkins/ init
git -C hugo-app-jenkins/ add .
git -C hugo-app-jenkins/ commit -m "Create Hugo site repository"
git -C hugo-app-jenkins/ remote add origin \
    https://github.com/${GITHUB_USERNAME}/hugo-app-jenkins.git
git -C hugo-app-jenkins/ push -u origin master

You now have your own repository populated with the Hugo site content.

Create the Jenkinsfile

Now let’s have a look at the Jenkinsfile that we will add to your repository:

#!groovy
podTemplate(label: 'pod-hugo-app', containers: [
    containerTemplate(name: 'hugo', image: 'smesch/hugo', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'html-proofer', image: 'smesch/html-proofer', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'kubectl', image: 'smesch/kubectl', ttyEnabled: true, command: 'cat',
        volumes: [secretVolume(secretName: 'kube-config', mountPath: '/root/.kube')]),
    containerTemplate(name: 'docker', image: 'docker', ttyEnabled: true, command: 'cat',
        envVars: [containerEnvVar(key: 'DOCKER_CONFIG', value: '/tmp/'),])],
        volumes: [secretVolume(secretName: 'docker-config', mountPath: '/tmp'),
                  hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')
  ]) {

    node('pod-hugo-app') {

        def DOCKER_HUB_ACCOUNT = 'smesch'
        def DOCKER_IMAGE_NAME = 'hugo-app-jenkins'
        def K8S_DEPLOYMENT_NAME = 'hugo-app'

        stage('Clone Hugo App Repository') {
            checkout scm
 
            container('hugo') {
                stage('Build Hugo Site') {
                    sh ("hugo --uglyURLs")
                }
            }
    
            container('html-proofer') {
                stage('Validate HTML') {
                    sh ("htmlproofer public --internal-domains ${env.JOB_NAME} --external_only --only-4xx")
                }
            }

            container('docker') {
                stage('Docker Build & Push Current & Latest Versions') {
                    sh ("docker build -t ${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:${env.BUILD_NUMBER} .")
                    sh ("docker push ${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:${env.BUILD_NUMBER}")
                    sh ("docker tag ${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:${env.BUILD_NUMBER} ${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:latest")
                    sh ("docker push ${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:latest")
                }
            }

            container('kubectl') {
                stage('Deploy New Build To Kubernetes') {
                    sh ("kubectl set image deployment/${K8S_DEPLOYMENT_NAME} ${K8S_DEPLOYMENT_NAME}=${DOCKER_HUB_ACCOUNT}/${DOCKER_IMAGE_NAME}:${env.BUILD_NUMBER}")
                }
            }

        }        
    }
}

Breakdown of the Jenkinsfile

Define a Pod template with four containers (hugo, html-proofer, kubectl & docker)
Secret kube-config will be mounted to /root/.kube in the kubectl container
Secret docker-config will be mounted to /tmp in the Docker container
Env variable DOCKER_CONFIG set to the same directory (/tmp) where the docker-config Secret is mounted
Set env variables for your Docker Hub account, the Docker image name and the K8s deployment name
Clone your GitHub repository
Build the Hugo site from your repository and output the static HTML to the default folder public
Run html-proofer against the public folder to test external links
Build a new Docker image with the updated site content, using the Dockerfile in the repository
Push the new Docker image to your Docker Hub account using both the build number and latest as the tags
Start a rolling update for the Kubernetes Deployment, using the new Docker image with the build number tag

Copy the Jenkinsfile to your GitHub repo folder:

cp github/hugo-app-jenkins/Jenkinsfile hugo-app-jenkins/Jenkinsfile

Update the DOCKER_HUB_ACCOUNT environment variable with your Docker Hub account:

vi hugo-app-jenkins/Jenkinsfile

    node('pod-hugo-app') {

        def DOCKER_HUB_ACCOUNT = 'smesch'

Create the Dockerfile

And below is the Dockerfile that will be used by the Jenkinsfile for the Docker image build:

FROM nginx:alpine
MAINTAINER Steven Eschinger <steven@kumorilabs.com>
COPY public /usr/share/nginx/html

Breakdown of the Dockerfile

Use the nginx:apline image as a base
Copy the static HTML content of the Hugo site from the public folder to the default NGINX folder

Copy the Dockerfile to your GitHub repo folder:

cp github/hugo-app-jenkins/Dockerfile hugo-app-jenkins/Dockerfile

Update MAINTAINER with your name and email:

vi hugo-app-jenkins/Dockerfile

...
MAINTAINER Steven Eschinger <steven@kumorilabs.com>
...

And finally, commit both files to GitHub:

git -C hugo-app-jenkins/ add .
git -C hugo-app-jenkins/ commit -m "Upload Jenkinsfile & Dockerfile"
git -C hugo-app-jenkins/ push -u origin master

Deploy Jenkins and the Hugo site

Let’s first deploy Jenkins. If you completed the previous lab, then you can update the JENKINS_DOCKER_IMAGE variable with your own image:

# Set the Jenkins Docker image name (leave as-is to use mine)
export JENKINS_DOCKER_IMAGE="smesch/jenkins-kubernetes-leader:2.46.2"

sed -i -e "s|image: .*|image: ${JENKINS_DOCKER_IMAGE}|g" \
    ./kubernetes/jenkins/jenkins-deploy.yaml

kubectl create -f ./kubernetes/jenkins/

deployment "jenkins-leader" created
persistentvolumeclaim "jenkins-leader-pvc" created
service "jenkins-leader-svc" created

Wait about a minute for the Service to create the AWS ELB and then create the DNS CNAME record in your Route 53 domain with a prefix of jenkins (e.g., jenkins.k8s.kumorilabs.com), using the dns-record-single.json template file in the repository:

# Set the DNS record prefix & the Service name and then retrieve the ELB URL
export DNS_RECORD_PREFIX="jenkins"
export SERVICE_NAME="jenkins-leader-svc"
export JENKINS_ELB=$(kubectl get svc/${SERVICE_NAME} \
       --template="{{range .status.loadBalancer.ingress}} {{.hostname}} {{end}}")

# Add to JSON file
sed -i -e 's|"Name": ".*|"Name": "'"${DNS_RECORD_PREFIX}.${DOMAIN_NAME}"'",|g' \
    scripts/apps/dns-records/dns-record-single.json
sed -i -e 's|"Value": ".*|"Value": "'"${JENKINS_ELB}"'"|g' \
    scripts/apps/dns-records/dns-record-single.json

# Create DNS records
aws route53 change-resource-record-sets \
    --hosted-zone-id ${DOMAIN_NAME_ZONE_ID} \
    --change-batch file://scripts/apps/dns-records/dns-record-single.json

And now let’s create the initial Deployment of the Hugo site:

kubectl create -f ./kubernetes/hugo-app/

deployment "hugo-app" created
service "hugo-app-svc" created

Wait about a minute for the Service to create the AWS ELB and then create the DNS CNAME record in your Route 53 domain with a prefix of hugo (e.g., hugo.k8s.kumorilabs.com), using the dns-record-single.json template file in the repository:

# Set the DNS record prefix & the Service name and then retrieve the ELB URL
export DNS_RECORD_PREFIX="hugo"
export SERVICE_NAME="hugo-app-svc"
export HUGO_APP_ELB=$(kubectl get svc/${SERVICE_NAME} \
       --template="{{range .status.loadBalancer.ingress}} {{.hostname}} {{end}}")

# Add to JSON file
sed -i -e 's|"Name": ".*|"Name": "'"${DNS_RECORD_PREFIX}.${DOMAIN_NAME}"'",|g' \
    scripts/apps/dns-records/dns-record-single.json
sed -i -e 's|"Value": ".*|"Value": "'"${HUGO_APP_ELB}"'"|g' \
    scripts/apps/dns-records/dns-record-single.json

# Create DNS records
aws route53 change-resource-record-sets \
    --hosted-zone-id ${DOMAIN_NAME_ZONE_ID} \
    --change-batch file://scripts/apps/dns-records/dns-record-single.json

Jenkins should now be reachable at the DNS name you just created (e.g., jenkins.k8s.kumorilabs.com) and if you are using my image, the credentials are below:

Username: admin
Password: jenkins

And the Hugo site should also be reachable (e.g., hugo.k8s.kumorilabs.com) and as we used the default 1.0 Docker image tag, the theme will be red:

Configure the GitHub Jenkins plugin

Before we create the Jenkins pipeline job, we first need to configure the GitHub Jenkins plugin.

Home Page >> Manage Jenkins >> Configure System >> Configure the following under the GitHub section:

Add GitHub Server >> GitHub Server
Click second Advanced button
Manage additional GitHub actions >> Convert login and password to token
Click From login and password radio button
Enter in your GitHub Login and Password
Click Create token credentials
Click Save

Go back to the same GitHub section in Home Page >> Manage Jenkins >> Configure System:

Select GitHub (https://api.github.com) auto generated token credentials from the Credentials dropdown box
Click second Advanced button
Select GitHub (https://api.github.com) auto generated token credentials from the Shared secret dropdown box
Click Save

Create the Jenkins pipeline for the Hugo site

Let’s now create the Jenkins pipeline job for the Hugo app, using the DNS name of the Hugo app for the job name:

Home Page >> New Item >> Enter Your Hugo App URL (e.g., hugo.k8s.kumorilabs.com) for Item name >> Click Pipeline >> Click Ok

Configure the following

Check the GitHub project box under General
Enter the project URL of your Hugo App GitHub repository for Project url (see below)

To get the project URL of your Hugo App GitHub repository:

echo "http://github.com/${GITHUB_USERNAME}/hugo-app-jenkins"

http://github.com/smesch/hugo-app-jenkins

Configure the following

Check the GitHub hook trigger for GITScm polling box under Build Triggers
Select Pipeline script from SCM from the Definition dropdown box under Pipeline
Select Git from the SCM dropdown box
Enter the full URL of your Hugo App GitHub repository under Repositories >> Repository URL (see below)

To get the full URL of your Hugo App GitHub repository:

echo "https://github.com/${GITHUB_USERNAME}/hugo-app-jenkins.git"

https://github.com/smesch/hugo-app-jenkins.git

Configure the following

Leave the rest as-is and click Save

Create the Secrets for Docker Hub and Kubectl authentication

Before we run the initial build, we first need to create the Secrets in your cluster for Docker Hub and Kubectl authentication. These Secrets will be made available to the Pod template we defined in the Jenkinsfile, which will allow it to push images to your Docker Hub account and to kick-off a rolling update in your cluster.

First let’s login to your Docker Hub account, which will generate the config.json file containing your Docker Hub authorization token:

docker login

Login with your Docker ID to push and pull images from Docker Hub. 
Username: smesch
Password:
Login Succeeded

Now let’s create the Secrets:

kubectl create secret generic docker-config \
        --from-file=$HOME/.docker/config.json

secret "docker-config" created

kubectl create secret generic kube-config \
        --from-file=$HOME/.kube/config

secret "kube-config" created

Test the Jenkins pipeline

To run the job, click Build Now. And then in the Build History area, click on the arrow next to the build number and click Console Output:

Jenkins - Console Output Hugo Pipeline Job

You will see the job running in real-time. An excerpt of the log is below:

Started by user admin
Obtained Jenkinsfile from git https://github.com/smesch/hugo-app-jenkins.git
[Pipeline] podTemplate
[Pipeline] node
kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28 is offline
Running on kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28 in /home/jenkins/workspace/hugo.k8s.kumorilabs.com
...
[Pipeline] { (Clone Hugo App Repository)
[Pipeline] checkout
Cloning the remote Git repository
Cloning repository https://github.com/smesch/hugo-app-jenkins.git
...
[Pipeline] { (Build Hugo Site)
Executing shell script inside container [hugo] of pod [kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28]
...
+ hugo --uglyURLs
Started building sites ...
Built site for language en:
5 regular pages created
14 other pages created
...
[Pipeline] { (Validate HTML)
Executing shell script inside container [html-proofer] of pod [kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28]
...
+ htmlproofer public --internal-domains hugo.k8s.kumorilabs.com --external_only --only-4xx
Running ["ImageCheck", "ScriptCheck", "LinkCheck"] on ["public"] on *.html... 

Checking 31 external links...
Ran on 5 files!

HTML-Proofer finished successfully.
...
[Pipeline] { (Docker Build & Push Current & Latest Versions)
Executing shell script inside container [docker] of pod [kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28]
...
+ docker build -t smesch/hugo-app-jenkins:2 .
Sending build context to Docker daemon  2.564MB
...
Successfully built c63b0b43cef2
...
+ docker push smesch/hugo-app-jenkins:2
The push refers to a repository [docker.io/smesch/hugo-app-jenkins]
...
+ docker tag smesch/hugo-app-jenkins:2 smesch/hugo-app-jenkins:latest
...
+ docker push smesch/hugo-app-jenkins:latest
The push refers to a repository [docker.io/smesch/hugo-app-jenkins]
...
[Pipeline] { (Deploy New Build To Kubernetes)
Executing shell script inside container [kubectl] of pod [kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28]
...
+ kubectl set image deployment/hugo-app-jenkins hugo-app-jenkins=smesch/hugo-app-jenkins:2
deployment "hugo-app-jenkins" image updated
...
[Pipeline] End of Pipeline
Finished: SUCCESS

If you run the job again and then switch to your terminal, you should be able to see a Pod being created in your cluster:

kubectl get pods

NAME                                                      READY     STATUS    RESTARTS   AGE
hugo-app-jenkins-3750695221-cbx2b                         1/1       Running   0          31m
hugo-app-jenkins-3750695221-cnc75                         1/1       Running   0          31m
hugo-app-jenkins-3750695221-kwfzq                         1/1       Running   0          31m
jenkins-leader-960176880-3bmvt                            1/1       Running   0          32m
kubernetes-4d96cfb4bddf455d90e08415f6e72517-1f444135f28   5/5       Running   0          30s

Now let’s setup the GitHub webhook, so that builds are automatically triggered when a commit is made to the GitHub repository:

Home Page >> Manage Jenkins >> Configure System >> Configure the following under the GitHub section:

Click second Advanced button
Click Re-register hooks for all jobs button
Click Save

We will now change the theme color to orange in the Hugo config file and then commit the change to your GitHub repository (you will be prompted for your GitHub credentials):

export HUGO_APP_TAG="orange"
sed -i -e 's|primary = .*|primary = "'"${HUGO_APP_TAG}"'"|g' \
    hugo-app-jenkins/config.toml
git -C hugo-app-jenkins/ pull
git -C hugo-app-jenkins/ commit -a -m "Set theme color to ${HUGO_APP_TAG}"
git -C hugo-app-jenkins/ push -u origin master

Within a few seconds of committing the change to GitHub, the Jenkins pipeline job should be started automatically. And when it completes, the theme color of the site will be orange:

You now have a fully functional continuous deployment pipeline setup in Jenkins.

Cleanup

Before proceeding to the next lab, delete the cluster and it’s associated S3 bucket:

Delete the cluster

Delete the cluster:

kops delete cluster ${CLUSTER_FULL_NAME} --yes

Delete the S3 bucket in AWS:

aws s3api delete-bucket --bucket ${CLUSTER_FULL_NAME}-state

In addition to the step-by-step instructions provided for each lab, the repository also contains scripts to automate some of the activities being performed in this blog series. See the Using Scripts guide for more details.

Next Up

In the next lab, Lab #8: Continuous Deployment with Travis CI and Kubernetes, we will go through the following:

Creating a continuous deployment pipeline in Travis CI for the Hugo site
Testing the pipeline in Travis CI

Introduction

Activities

Prerequisites

Implementation

Deploy a new cluster

Create the Hugo site

Create the GitHub repository for the Hugo site

Create the Jenkinsfile

Create the Dockerfile

Deploy Jenkins and the Hugo site

Configure the GitHub Jenkins plugin

Create the Jenkins pipeline for the Hugo site

Create the Secrets for Docker Hub and Kubectl authentication

Test the Jenkins pipeline

Cleanup

Delete the cluster

Next Up

Other Labs in the Series

Categories

Tags

Getting To Know K8s | Lab #7: Continuous Deployment with Jenkins and Kubernetes

Introduction

Activities

Prerequisites

Implementation

Deploy a new cluster

Create the Hugo site

Create the GitHub repository for the Hugo site

Create the Jenkinsfile

Create the Dockerfile

Deploy Jenkins and the Hugo site

Configure the GitHub Jenkins plugin

Create the Jenkins pipeline for the Hugo site

Create the Secrets for Docker Hub and Kubectl authentication

Test the Jenkins pipeline

Cleanup

Delete the cluster

Next Up

Other Labs in the Series

Categories

Tags