Jenkins Workshop

Pipelines & Blue Ocean 101

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License .

Hello!

Olivier VERNIN

overnin

Damien DUPORTAL

damien
Presentation CloudBees

And you?

commitstrip

  • Developers?

  • Who is doing Continuous Integration/Delivery/Deployment?

    • Which tools?

      • Jenkins "Old School" ? / "New Generation" ?

      • GitLab ? TeamCity ? TravisCI ? Bamboo ?

      • Something Else?

Workshop

Goals

  • It is a "101" (starter) about:

    • Jenkins (Declarative) Pipelines

    • Blue Ocean

  • It is a Workshop: learn by "doing"

    • We start from scratch

    • Come back home with a fully working example

  • Sources available on Github:

At Home: Getting the Workshop Lab as a local VM

  • You need Vagrant (2.1.0+) and VirtualBox (5.2.12+)

  • Run the following commands to start the local VM:

# Getting the VM template (1 time only)
vagrant box add jenkins-lab-demo https://bit.ly/2HvmzGu
# Initialize the environment (1 time only)
mkdir ~/jenkins-lab-demo && cd ~/jenkins-lab-demo
vagrant init -m -f jenkins-lab-demo
# Start the lab
vagrant up

On-Site: Getting the Workshop Lab as a Cloud VM

  • We provide you a set of Cloud VMs: only web browser is required

  • Access the "Instance Sheet Page", put your name on the "Name" column to allocate the instance for yourself, and click on the "Instance URL" link on the same line.

  • Welcome to the Lab Home Page!

Accessing Customized Slides

  • Each instance hosts a copy of this slide-deck, with customized links

  • If you click the link "Workshop Slides", a new tab will open, to this slide-deck

Accessing Source Code

Accessing Jenkins

Jenkins and Blue Ocean

Jenkins

jenkins logo

  • Task orchestrator

  • Open-Source software written in Java

  • One of the first and most popular Continous Integration Engine (2004)

  • Plugin based

Jenkins Pipelines

pipeline

  • A tool to express your software pipeline with a DSL

  • Defined into a text file: Jenkinsfile

  • "Pipeline-as-code": stored into your SCM

  • != "Job DSL" (different areas)

Benefits of Jenkins Pipelines

  • Follows from your code workflows: branches, pull request, code review, even trunk-based

  • Reduces the number of jobs

  • A single execution ("build") can run on multiple machines

  • Durable: survives a Jenkins restart

Getting started with Pipeline

i have no idea what i am doing
BlueOcean

Blue Ocean

  • A new developer experience for using Jenkins

  • A modern GUI written in React, in parallel of the "Classic UI"

  • Pipeline-centric

  • Install it with a plugin

Your First Pipeline

Blue Ocean

  • We want to create a Pipeline from scratch using Blue Ocean.

  • Browse to your Jenkins instance and switch to Blue Ocean:

Follow along: Initialize Your First Pipeline 1/4

  • Follow the presenter instructions to create a new Pipeline:

    • Stored in Git

    • Get the SSH URL from the git server

      • ssh://git@gitserver:5022/butler/demoapp.git

    • Jenkins is already allowed to access the source code:

      • Nothing to do for SSH Key

Blue Ocean Pipeline Editor

  • Provides a Pipeline’s graphical editor in Blue Ocean, with full round trip with SCM:

    • Scaffold your Pipeline, Save it, Execute it.

  • Supports existing Declarative Pipeline, as starting from scratch

  • Requires a compatible SCM (Git over SSH, Github, BitBucket, etc.)

Follow along: Initialize Your First Pipeline 2/4

  • Blue Ocean cannot find any Jenkinsfile

    • It opens automatically the Pipeline Editor for you

  • A red message appears saying: "a stage is needed":

    • Create a stage named Build by clicking the big "+"

  • Each stage (virtual) is composed of "steps" (implementation)

    • "At least one step is required"

Follow along: Initialize Your First Pipeline 3/4

  • We want the stage Build to have 2 steps:

    • Print the message Building…​

    • Execute the shell script ./scripts/build.sh

  • Time to run this Pipeline: click the "Save" button

    • Add a meaningful Description (commit message)

    • Commit to 'master'

    • Clic Save & Run, and see the build kicking off

Follow-along: Initialize Your First Pipeline 4/4

  • Take time to browse the build page in Blue Ocean

    • Click on the stage, unfold the logs…​

  • Click the "Edit" (pen) button to go back to the editor, and use CTRL + S (On Mac: CMD + S), to switch to textual description:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'Building...'
        sh './scripts/build.sh'
      }
    }
  }
}

Exercise: Add 2 New Stages to your First Pipeline

  • Goal: add 2 new stages to your Pipeline:

    Integration Tests

    runs the script ./scripts/integration-tests.sh

    Deploy

    runs the script ./scripts/deploy.sh

  • Use the Blue Ocean Pipeline Editor:

    • Click on the "Edit" button (pen’s icon)

    • Use the Build stage as example

Solution: Add 2 New Stages to your First Pipeline

  • Check the file Jenkinsfile into the SCM (Gitea Git Server)

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'Building...'
        sh './scripts/build.sh'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Pipeline: Tests and Webhooks

Follow-along: Test Reports with JUnit 1/3

  • We want to report test results to Jenkins

  • Let’s start with Unit Test’s results:

    • They are in "Junit" format, generated during the Build stage

    • XML files located in:

      target/surefire-reports/

    • Jenkins’s "Junit" plugin is installed by default

Follow-along: Test Reports with JUnit 2/3

  • Open the Pipeline in Blue Ocean Editor

  • Edit the stage Build

  • Remove the Print Message step

  • Add a new Archive JUnit-formatted test results step

    • Set the field TestResults to target/surefire-reports/*.xml

  • Save the pipeline and check the build

Follow-along: Test Reports with JUnit 3/3

  • When the build finished, check the "Tests" section

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Exercise: Integration Tests Report

  • We also want the integration test’s reports

    • Generated during the Integration Tests stage

    • Still using JUnit, but XML files are generated in:

      target/failsafe-reports/

  • Expected status of the build: UNSTABLE

Solution: Integration Tests Report

  • What are the changes between UNSTABLE and SUCCESS?

    • Color, Test failure notification badge, etc.

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
        junit 'target/failsafe-reports/*.xml'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Continuous Integration (CI)

fail fast continuous integration

  • Each code integration is validated by an automated build

  • Code is integrated often, at least daily, to be a non-event

  • Feedback loop to close the continous cycle

Faster Feedbacks using Webhooks

  • We want builds to start when:

    • A Pipeline is saved from Blue Ocean ⇒ OK

    • New code is added in the SCM ⇒ ??

  • A "Webhooks" is already configured:

    • When new code is added into the SCM,

    • Then the SCM server sends a request to notify the event

    • And Jenkins reacts to this event by indexing the repository, and eventually starts new builds

Exercise: Fix the build

  • We are doing Continous Integration!

    • The build MUST be fixed as top priority

  • Identify the name of the failing test

    • TIP: Use Blue Ocean’s "Test" tab

  • Use the SCM (the Gitea Web UI) to edit the code

    • Tests are stored into src/test/java/hello/

    • Each time the code is changed, a build is started

Solution: Fix the build

Going further…​

Agents

What is an Agent?

  • A process running on a remote node, ready to execute tasks on behalf of the Jenkins master

    • Formerly known as slave

orchestra

Benefits of Agents

  • Specific Agent Roles:

    • Custom operating system (Windows, Linux)

    • Specialized hardware (Macs, Solaris, Arm CPUs, etc.)

  • Horizontal Scalability of builds

    • More builds? More machines!

  • Diminution of "Blast Radius"

    • An agent goes down? Another takes the load

    • Jenkins Master goes down (OOM, Restart): no impact

Agent and Pipeline

  • A pipeline’s build is executed on an agent

    • The keyword agent must be specified

  • Some stages can be executed on different agents:

    • E.g. Build on Linux, Test on Linux and Windows

    • The keyword agent can be specified per stage

Follow-along: Specify Agent for our Pipeline 1/2

  • Edit the Pipeline with the Blue Ocean Editor

  • Access the "Pipeline Settings" on the right section

    • TIP: If not visible, click on the "Start" node

  • Change the agent any to a node with the label maven3-jdk8

    • "any" was picking randomly an available agent

    • Note the other options

  • Save and run the Pipeline

Follow-along: Specify Agent for our Pipeline 2/2

  • Search for occurrences of "Running on" in the Pipeline’s log

    • The file pipeline.log can be found in "Artifacts"

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent {
    node {
      label 'maven-jdk8'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
        junit 'target/failsafe-reports/*.xml'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Exercise: Execute Deploy stage in "production"

  • Goal: the Stage Deploy has to be run in (fake) "production"

  • Tools: there is an agent with the label "production"

  • Change the Pipeline to have only the Deploy stage running on this agent

Solution: Execute Deploy stage in "production"

  • Validate using the occurrences of Running on in the pipeline.log

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent {
    node {
      label 'maven-jdk8'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
        junit 'target/failsafe-reports/*.xml'
      }
    }
    stage('Deploy') {
      agent {
        node {
          label 'production'
        }
      }
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Reusing Binary: Challenge

  • Pipeline efficiency and security rule: build once, fingerprint and then reuse

  • Challenge: workspace is NOT shared across agents.

    • A new workspace is allocated for each stage

Reusing Binary: Solution

  • Pipeline can store artifacts from stage to stage

    • Assumption: building a binary is 1 order of magnitude slower than copying it on the network

  • Two types of artifact storages, both using a copy-to-master:

    • Permanent storage with archiveArtifacts.

    • Ephemeral storage with stash (and unstash), deletion when the build stops.

Follow-along: Reuse Jar File 1/3

  • Goal: we want the stages Integration Test and Deploy to reuse the folder target generated during the stage Build

Follow-along: Reuse Jar File 2/3

  • Edit the stage Build and add a new step:

    • Type of the new step: "Stash some files to be used later in the build"

    • Name: generated-artifacts

    • Include: target/*/ (the folder target/ and all its content)

  • For both Integration Tests and Deploy stages:

    • Add a new step of type "Restore files previously stashed"

    • Set the Name to generated-artifacts

    • Move these steps at the beginning of each stage

Follow-along: Reuse Jar File 3/3

  • Validate using the logs of the Deploy stage (comparing with previous builds)

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent {
    node {
      label 'maven3-jdk8'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
        stash(name: 'generated-artifacts', includes: 'target/**/*')
      }
    }
    stage('Test') {
      steps {
        unstash 'generated-artifacts'
        sh './scripts/integration-tests.sh'
        junit 'target/failsafe-reports/*.xml'
      }
    }
    stage('Deploy') {
      agent {
        node {
          label 'production'
        }
      }
      steps {
        unstash 'generated-artifacts'
        sh './scripts/deploy.sh'
      }
    }
  }
}

Exercise: Parallel Integration Tests

  • Goal: Run Integration Tests for both Java 9 and 8

  • Tools: you have 2 available "java" agents with the respective labels jdk9 and jdk8

  • Edit the Pipeline to execute, in parallel, the integration tests on each of these agents

    • Keep the stage Integration Tests as parent stage

    • Name the "sub-stages" IT Java 9 and IT Java 8

    • Don’t forget to unstash from Build and to publish test reports.

Solution: Parallel Integration Tests

  • Validate using Test reports (compared to previous builds), and build logs

  • Let’s have a look to the expected Pipeline:

pipeline {
  agent {
    node {
      label 'maven3-jdk8'
    }
  }
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
        stash(name: 'generated-artifacts', includes: 'target/**/*')
      }
    }
    stage('Test') {
      parallel {
        stage('IT Java 8') {
          agent {
            node {
              label 'jdk8'
            }
          }
          steps {
            unstash 'generated-artifacts'
            sh './scripts/integration-tests.sh'
            junit 'target/failsafe-reports/*.xml'
          }
        }
        stage('IT Java 9') {
          agent {
            node {
              label 'jdk9'
            }
          }
          steps {
            unstash 'generated-artifacts'
            sh './scripts/integration-tests.sh'
            junit 'target/failsafe-reports/*.xml'
          }
        }
      }
    }
    stage('Deploy') {
      agent {
        node {
          label 'production'
        }
      }
      steps {
        unstash 'generated-artifacts'
        sh './scripts/deploy.sh'
      }
    }
  }
}