Certified Jenkins Enginee

Certified Jenkins Engineer (CJE): https://github.com/bmuschko/cje-crash-course

Jenkins

Jenkins is not a build system, it is a structured model. Other interesting project: Tekton , Jenkins X (k8s involved)

Installing Jenkins, must have compatible openjdk installed.

1 2	## can use war file: java -jar jenkins.war

or using rpm install for centos/redhat, then using systemctl to start/enable Jenkins https://www.jenkins.io/doc/book/installing/#red-hat-centos

1 2	## you will see the default jenkins port is 8080 ps aux \| grep jenkinx

then you can open the web interface by <node ip>:8080 . 如果发现是中文版，可能是浏览器的语言设置出了问题，改一下chrome的语言设置为English即可.

If wizard install failed with some plugins, you can fix this later in Manage Plugins .

Jenkins use file system to store everything, if using systemd, the configuration is in /var/lib/jenkins . You can backup this folder, or if you want to wipe it out, then run systemctl restart jenkins , then jenkins goes back to init state.

Even Jenkins UI to create project is not needed, you can mkdir and files in the Jenkins working directory, then go to Manage Jenkins -> Reload Configuration from Disk .

遇到一件很神奇的事情，有次Jenkins的Credentials配置消失了。。。重启也不见恢复，后来我直接stop daemon, 清空workspace下所有文件，再次重启初始化，就恢复了。后来想想我应该是在配置Credentials时把配置改错了，可以通过 Manage Jenkins -> Configure Credentials 改回去。

Creating app build

freestyle project -> pipeline (series of freestyle project), freestyle project is not recommended.

Jenkins Workspace , you can see it in console output or click Workspace icon in your project dashboard. Everything running is in project workspace. Every build will override the previous one. You can use tar command to backup and restore this workspace, or clean workspace.

注意, 如果在Jenkins configuration中直接使用pipeline script 而不是 SCM, 是不会创建workspace的。

To store the build artifact, use Post-build Action in configure. for example, you want to archive some jar or zip files. then after build is done, these archives will show in the build page.

Build trend

Right to Build History list, there is a trend button, click it will see the build time history statistics and distribution.

Testing and Continuous integration

Now start the pipeline job type. After creating a pipeline job, you will see pipeline syntax button in the page bottom, it contains necessary resources to start. You can also use Copy from to copy a pipeline configure from another, for quick start.

Add slave nodes

Manage Jenkins -> Manage Nodes and Clouds To add slaves, usually use SSH to launch agent nodes. (如果node没有被发现，会显示错误，根据错误指示排查问题即可)

Before adding a slave node to the Jenkins master we need to prepare the node. We need to install Java on the slave node. Jenkins will install a client program on the slave node.

To run the client program we need to install the same Java version we used to install on Jenkins master. You need to install and configure the necessary tool in the slave node.

1	yum install -y java-1.8.0-openjdk

When configure add node agent, Host Key Verification Strategy:

Host Key Verification for SSH Agents

Pipeline steps

This is scripted pipeline syntax, not recommended! Please use declarative pipeline directives! what are the differences between them: https://www.jenkins.io/doc/book/pipeline/#pipeline-syntax-overview

如果直接在Jenkins configure UI中设置Jenkins file，则常用的Steps (in snippet generator):

node: Allocate node Jenkins use master <-> agent model , you can configure tasks to be executed in agent node.

stage: stage

stash: stash some files to be used later in build

unstash: restore files previous stashed

parallel: execute in parallel (如果注册了多个slave nodes，则parallel会在上面执行并行的任务，一般用在测试的时候，比如测试不同的环境和配置, see declarative pipeline demo code below)

git: Git

dir: Change Current Directory

sh: Shell Script

step: General Build Step

emailext: Extended Email

Triggering auto build

在pipeline configure中有 Builder Triggers 可以选择:

Build after other projects are built

Build periodically

Poll SCM

Disable this project

Quiet period

Trigger builds remotely

Email Notification

Using emailext: Extended Email ，可以用groovy函数包装，传入email的subject以及内容再调用。

Managing plugins

Manage Jenkins -> Manage Plugins , then you can select and install plugin in Available section. For example:

Pipeline (如果这个没安装，则不会在UI中显示pipeline的动态流程图)

Html publisher (常用于发布unit test后的html report，这些html文件其实是被相关test生成好的, publisher then renders it)

publishHTML([allowMissing: true,
            alwaysLinkToLastBuild: true,
                keepAll: true,
                reportDir: "$WORKSPACE/cognitive-designer-api/DSJsonApiServletJUnitTests/build/reports/tests/payloadtests",
                reportFiles: 'index.html',
                reportName: 'Payload Test',
                reportTitles: ''])

Green Balls (show green color for success)

Blue Ocean (embedded site with new Jenkins UI)

Job DSL: Allowing jobs to be defined in a programmatic form in a human readable file.

Pipeline compatible plugins: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md

在初始化设置Jenkins的时候，有可能有plugins安装失败，可以自己在 Manage plugin 中安装，然后restart Jenkins (关于restart Jenkins请在没有job运行的情况下进行，不同的安装方式restart的方法不同，或者在安装plugin的时候选择restart jenkins after install), for example: systemctl restart jenkins , 这可以消除控制面板上的plugin failure警告。。

Continuous delivery

In Blue Ocean , you can run multiple builds in parallel. if more than one builds run in the same agent, the workplace path is distinguished by suffix (@count number). 但是能不能run multiple builds in one agent depends on how you design your pipeline and tasks.

Blue Ocean 中的UI对parallel的显示也很直观，方便查看。

Trigger builds remotely

Set pipeline can be triggered remotely by URL, also optionally set pipeline trigger token in pipeline configure UI (can be empty).

You also need to know the user token, set from current user profile menu, you must keep the your user token to somewhere, for example, store it in credential secret text. So you can refer the token for example:

1 2	TRIGGER_USER = "chengdol.example.com" TRIGGER_USER_TOEKN = credentials('<token id>')

Then in the upstream pipeline script, trigger other pipeline by running curl command:

## can be http or https connection
## --user is the jenkin user and its token or password
## token=${PIPELINE_TRIGGER_TOKEN} can be ignored if it's empty
curl --user ${TRIGGER_USER}:${TRIGGER_USER_TOEKN} --request POST http/https://<url>/job/${PIPELINE_NAME}/buildWithParameters?token=${PIPELINE_TRIGGER_TOKEN}\\&para1=val1\\&&para2=val2

You don’t need to specify all parameters in URL, the parameters default values will be used if they are not specified in the URL.

Notice that para1 and para2 must exist in parameters section of the triggered pipeline, otherwise you cannot use them. So far, based on testing, I can pass string, bool and file parameter types.

Check Status of Another pipeline

reference: https://gist.github.com/paul-butcher/dc68adc1c05ca970158c18206756dab1

1	curl --user ${LOGIN_USER}:${LOGIN_USER_TOEKN} --request GET http/https://<url>/job/${PIPELINE_NAME}/<build number>/api/json

Then you can parse the json returned: artifacts -> result: SUCCESS, FAILURE, ABORTED

Flyweight executor

Flyweight executor reside in Jenkins master, used to execute code outside of node allocation. Others are heavyweight executors. Flyweight executor will not be counted into executor capacity.

For example, we use flyweight executor for pause, in Jenkins script: https://stackoverflow.com/questions/44737036/jenkins-pipeline-with-code-pause-for-input

1 2	## see below declarative pipeline demo code input "waiting for approval, move to staging stage..."

The job will be paused, you can go to Paused for Input to decide what to do next: proceed or abort. (In Blue Ocean , the pause interface is more clear)

Declarative pipeline

Please use Groovy style to wirte declarative pipeline!

github demo: https://github.com/sixeyed/jenkins-pipeline-demos

Declarative Pipelines: https://www.jenkins.io/doc/book/pipeline/syntax/

这里有关于declarative pipeline的介绍视频, Jenkins file lives in source control! https://www.jenkins.io/solutions/pipeline/ ,

Using Blue Ocean to setup pipeline from github need personal access token (must check out repo and user options): https://www.youtube.com/watch?v=FhDomw6BaHU

In Jenkins UI, go to pipeline syntax then declarative directive generator , it will help you generate pipeline code for declarative pipeline: https://www.jenkins.io/doc/book/pipeline/getting-started/#directive-generator

Jenkins parameters variables can be accessed by both params and env prefix, see this issue .

This is just a basic structure demo:

pipeline {
  // default agent specify
  agent any
  // pipeline level env var, global scope
  environment {
    // you can put release number here
    // referred by env.RELEASE
    RELEASE = '1.1.3'
  }
  // can have multiple stages
  stages {
    // list tool version
    stage('Audit tools') {
      steps {
          sh '''
            git version
            docker version
          '''
      }
    }
    stage('Build') {
      // agent specify
      agent any
      // stage level env var, stage scope
      environment {
        USER = 'root'
      }
      steps {
        echo "this is Build stage"
        // executable in your repo
        sh 'chmod +x ./build.sh'
        // 把jenkins中一个名为api-key的密匙的值 放入 API_KEY这个环境变量中
        // 且这个API_KEY仅在block中可见
        withCredentials([string(credentialsId: 'api-key', variable: 'API_KEY')]) {
          sh '''
              ./build.sh
          '''
          }
      }
    } 
    // can have different type
    stage('Test') {
      environment {
        LOG_LEVEL = "INFO"
      }
      // parallel tasks
      parallel {
        // they can be running on different agent
        // depends on you agent setting
        stage('test1')
        {
          steps {
            // show current stage name test1
            echo "parallel ${STAGE_NAME}"
            // switch to ./src directory
            dir('./gradle') {
              sh '''
              ./gradlew -p xxx test1
              '''
            }
          }
        }
        stage('test2')
        {
          steps {
            echo "parallel ${STAGE_NAME}"
          }
        }
        stage('test3')
        {
          steps {
            echo "parallel ${STAGE_NAME}"
          }
        }
      }
    }
    stage('Deploy') {
      // waiting for user input before deploying
      input {
        message "Continue Deploy?"
        ok "Do it!"
        parameters {
          string(name: 'TARGET', defaultValue: 'PROD', description: 'target environment')
        }
      }
      steps {
        echo "this is Deploy with ${env.RELEASE}"
        // groovy code block
        // potential security hole, jenkins will not make it easy for you
        script {
          // you need to approve use of these class/method
          if (Math.random() > 0.5) {
              throw new Exception()
          }
          // you can use try/catch block for security reason
        }
        // if fail, this wouldn't get executed
        // write 'passed' into file test-results.txt
        writeFile file: 'test-results.txt', text: 'passed'
      }
    } 
  }
  
  post {
    // will always be executed
    always {
      echo "prints whether deploy happened or not, success or failure."
    }
    // others like: success, failure, cleanup, etc
    success {
      // archive files
      archiveArtifacts 'test-results.txt'
      // slack notifation
      slackSend channel: '#chengdol-private',
                   message: "Release ${env.RELEASE}, success: ${currentBuild.fullDisplayName}."
    }
    failure {
         slackSend channel: '#chengdol-private',
                   color: 'danger',
                   message: "Release ${env.RELEASE}, FAILED: ${currentBuild.fullDisplayName}."
      }
  }
}

if you don’t want to checkout SCM in stages that run in same agent, you can use this option:

1
2
3

options {
  skipDefaultCheckout true
}

Configure slack

This is a slightly out-of-date video, 其实在各自pipeline中可以自己单独配置token以及选择channel。 https://www.youtube.com/watch?v=TWwvxn2-J7E

First install slack notifaction plugin. Then go to Manage Jenkins -> Configure System , scroll down to bottom you will see slack section, see question mark for explanation.

Then go to your target slack channel, select Add an app , search Jenkins CI , then add it to slack, follow the instructions to get the secret token, add this token to Jenkins credentials and use it in above slack configuration.

After all set, try Test connection , you will see message in your slack channel.

Reusable

Reusable functions and libraries are written in Groovy . https://www.eficode.com/blog/jenkins-groovy-tutorial

Let’s see some demos:

pipeline {
    agent any
    // options
    parameters {
        booleanParam(name: 'RC', defaultValue: false, description: 'Is this a Release Candidate?')
    }
    environment {
        VERSION = "0.1.0"        
        VERSION_RC = "rc.2"
    }
    stages {
        stage('Audit tools') {                        
            steps {
                // call function
                auditTools()
            }
        }
        stage('Build') {
            environment {
                // call function
                VERSION_SUFFIX = getVersionSuffix()
            }
            steps {
              echo "Building version: ${VERSION} with suffix: ${VERSION_SUFFIX}"
              sh 'dotnet build -p:VersionPrefix="${VERSION}" --version-suffix "${VERSION_SUFFIX}" ./m3/src/Pi.Web/Pi.Web.csproj'
            }
        }
        stage('Unit Test') {
            steps {
              // switch directory
              dir('./m3/src') {
                sh '''




    

                    dotnet test --logger "trx;LogFileName=Pi.Math.trx" Pi.Math.Tests/Pi.Math.Tests.csproj
                    dotnet test --logger "trx;LogFileName=Pi.Runtime.trx" Pi.Runtime.Tests/Pi.Runtime.Tests.csproj
                '''
                mstest testResultsFile:"**/*.trx", keepLongStdio: true
              }
            }
        }
        stage('Smoke Test') {
            steps {
              sh 'dotnet ./m3/src/Pi.Web/bin/Debug/netcoreapp3.1/Pi.Web.dll'
            }
        }
        stage('Publish') {
            // condition
            when {
                expression { return params.RC }
            } 
            steps {
                sh 'dotnet publish -p:VersionPrefix="${VERSION}" --version-suffix "${VERSION_RC}" ./m3/src/Pi.Web/Pi.Web.csproj -o ./out'
                archiveArtifacts('out/')
            }
        }
    }
}

// groovy methods, can run straight groovy code
def String getVersionSuffix() {
    if (params.RC) {
        return env.VERSION_RC
    } else {
        return env.VERSION_RC + '+ci.' + env.BUILD_NUMBER
    }
}

def void auditTools() {
    sh '''
        git version
        docker version
        dotnet --list-sdks
        dotnet --list-runtimes
    '''
}

Shared library

Demo structure and code: https://github.com/sixeyed/jenkins-pipeline-demo-library Invoking shared library at head of jenkins file:

// this is dynamic reference, explicitly specify the library in jenkins file
library identifier: 'jenkins-pipeline-demo-library@master', retriever: modernSCM(
        [$class: 'GitSCMSource',
        remote: 'https://github.com/sixeyed/jenkins-pipeline-demo-library.git',
        // if the repo is private, you can have credential here
        credentialsId: '<credential id>'])

pipeline {
    agent any
    stages {
        stage('Audit tools') { 
            environment {
                // pass parameters as map
                VERSION_SUFFIX = getVersionSuffix rcNumber: env.VERSION_RC, isReleaseCandidate: params.RC
            }
            steps {
                auditTools()
            }
        }
    }
}

You can add Global Pipeline Libraries in Configure Jenkins for any pipeline use. 这种情况下，可以设置默认的shared library，然后在jenkins file中直接调用相关函数。

Shared pipelines

You can put the shared pipeline into a shared library, for example:

library identifier: 'jenkins-pipeline-demo-library@master', 
        retriever: modernSCM([$class: 'GitSCMSource', remote: 'https://github.com/sixeyed/jenkins-pipeline-demo-library.git'])

crossPlatformBuild repoName: 'sixeyed/pi-psod-pipelines',
                   linuxContext: 'm4', 
                   windowsContext: 'm4'

Shared library is under vars folder. In Groovy, we can add a method named call to a class and then invoke the method without using the name call , crossPlatformBuild is actually the file name, inside file there is a call method.

Multi-branch pipeline

https://www.jenkins.io/doc/book/pipeline/multibranch/ Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.

Orphaned item strategy, for deleted branch, you can discard or reserve it.

Pipeline development tools

Validating pipeline syntax

First enable anonymous read access in Configure Global Security : https://www.jenkins.io/doc/book/pipeline/development/#linter

Issue curl command:

1 2	## if not success, it will show you the overall problems with your jenkins file curl -X POST -F "jenkinsfile=<[jenkins file path]" http://<IP>:8080/pipeline-model-converter/validate

Visual Studio code has Jenkins linter plugin, you need to configure it with linter url.

Restart or replay

In every build interface, restart from stage , you can select which stage to restart (sometimes stage may fail due to external reason), replay , you can edit your jenkins file and library then rerun, the changes only live in current build (after succeed, check in your updates to source control).

Unit test

https://github.com/jenkinsci/JenkinsPipelineUnit

Supports running pipelines and library methods

Can mock steps and validate calls

Jenkins with Docker

学习步骤: 首先是agent 使用docker，然后master + agent 都使用docker, 最后交由K8s去管理。

这块很有意思，加入容器后就太灵活了，只要有build agent支持docker且已安装，则jenkins就可以把container运行在之上。 https://www.jenkins.io/doc/book/pipeline/docker/

https://hub.docker.com/r/jenkins/jenkins you can parallelly run several jenkins version in one machine, for purposes like testing new features, testing upgrade, so on and so forth. But you may need to customize the jenkins docker image and expose to different port.

containers as build agents

customizing the build container

using the docker pipeline plugin

Jenkins master and slave with docker: https://medium.com/@prashant.vats/jenkins-master-and-slave-with-docker-b993dd031cbd

From agent syntax: https://www.jenkins.io/doc/book/pipeline/syntax/#agent

agent {
    docker {
        image 'myregistry.com/node'
        // can pass argument to docker run
        args  '-v /tmp:/tmp'
        // the node must pre-configured to have docker
        label 'my-defined-label'
        // optional set the registry to pull image
        registryUrl 'https://myregistry.com/'
        registryCredentialsId 'myPredefinedCredentialsInJenkins'
    }
}

If there is no label option, Jenkins will dynamically provisioned on a node and it will fail if no docker installed, you can set docker label to filter: https://www.jenkins.io/doc/book/pipeline/docker/#specifying-a-docker-label

I got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: https://stackoverflow.com/questions/47854463/docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socke

还解决了一个权限的问题，在实验中container默认用户是 jenkins ，没有root权限无线运行container内部的程序，解决办法是 args '-u root' 在上面的配置中。

此外jenkins会把workspace注入到container中，通过环境变量查找:

1 2	sh 'printenv' sh 'ls -l "$WORKSPACE"

Additionally, you can use agent with Dockerfile and install Docker Pipeline plugin.

1. Certified Jenkins Enginee
2. Jenkins
3. Declarative pipeline
4. Pipeline development tools
5. Jenkins with Docker