相关文章推荐

Vagrant Jenkins server git repo for testing purpose: https://github.com/chengdol/vagrant-jenkins

Certified Jenkins Enginee

Certified Jenkins Engineer (CJE): https://github.com/bmuschko/cje-crash-course

Jenkins

Jenkins is not a build system, it is a structured model. Other interesting project: Tekton , Jenkins X (k8s involved)

Installing Jenkins, must have compatible openjdk installed.

1
2
## can use war file:
java -jar jenkins.war

or using rpm install for centos/redhat, then using systemctl to start/enable Jenkins https://www.jenkins.io/doc/book/installing/#red-hat-centos

1
2
## you will see the default jenkins port is 8080
ps aux | grep jenkinx

then you can open the web interface by <node ip>:8080 . 如果发现是中文版,可能是浏览器的语言设置出了问题,改一下chrome的语言设置为English即可.

If wizard install failed with some plugins, you can fix this later in Manage Plugins .

Jenkins use file system to store everything, if using systemd, the configuration is in /var/lib/jenkins . You can backup this folder, or if you want to wipe it out, then run systemctl restart jenkins , then jenkins goes back to init state.

Even Jenkins UI to create project is not needed, you can mkdir and files in the Jenkins working directory, then go to Manage Jenkins -> Reload Configuration from Disk .

遇到一件很神奇的事情,有次Jenkins的Credentials配置消失了。。。重启也不见恢复,后来我直接stop daemon, 清空workspace下所有文件,再次重启初始化,就恢复了。后来想想我应该是在配置Credentials时把配置改错了,可以通过 Manage Jenkins -> Configure Credentials 改回去。

Creating app build

freestyle project -> pipeline (series of freestyle project), freestyle project is not recommended.

Jenkins Workspace , you can see it in console output or click Workspace icon in your project dashboard. Everything running is in project workspace. Every build will override the previous one. You can use tar command to backup and restore this workspace, or clean workspace.

注意, 如果在Jenkins configuration中直接使用pipeline script 而不是 SCM, 是不会创建workspace的。

To store the build artifact, use Post-build Action in configure. for example, you want to archive some jar or zip files. then after build is done, these archives will show in the build page.

Build trend

Right to Build History list, there is a trend button, click it will see the build time history statistics and distribution.

Testing and Continuous integration

Now start the pipeline job type. After creating a pipeline job, you will see pipeline syntax button in the page bottom, it contains necessary resources to start. You can also use Copy from to copy a pipeline configure from another, for quick start.

Add slave nodes

Manage Jenkins -> Manage Nodes and Clouds To add slaves, usually use SSH to launch agent nodes. (如果node没有被发现,会显示错误,根据错误指示排查问题即可)

Before adding a slave node to the Jenkins master we need to prepare the node. We need to install Java on the slave node. Jenkins will install a client program on the slave node.

To run the client program we need to install the same Java version we used to install on Jenkins master. You need to install and configure the necessary tool in the slave node.

1
yum install -y java-1.8.0-openjdk

When configure add node agent, Host Key Verification Strategy:

  • Host Key Verification for SSH Agents
  • Pipeline steps

    This is scripted pipeline syntax, not recommended! Please use declarative pipeline directives! what are the differences between them: https://www.jenkins.io/doc/book/pipeline/#pipeline-syntax-overview

    如果直接在Jenkins configure UI中设置Jenkins file,则常用的Steps (in snippet generator):

    node: Allocate node Jenkins use master <-> agent model , you can configure tasks to be executed in agent node.

    stage: stage

    stash: stash some files to be used later in build

    unstash: restore files previous stashed

    parallel: execute in parallel (如果注册了多个slave nodes,则parallel会在上面执行并行的任务, 一般用在测试的时候,比如测试不同的环境和配置, see declarative pipeline demo code below)

    git: Git

    dir: Change Current Directory

    sh: Shell Script

    step: General Build Step

    emailext: Extended Email

    Triggering auto build

    在pipeline configure中有 Builder Triggers 可以选择:

  • Build after other projects are built
  • Build periodically
  • Poll SCM
  • Disable this project
  • Quiet period
  • Trigger builds remotely
  • Email Notification

    Using emailext: Extended Email ,可以用groovy函数包装,传入email的subject以及内容再调用。

    Managing plugins

    Manage Jenkins -> Manage Plugins , then you can select and install plugin in Available section. For example:

    Pipeline (如果这个没安装,则不会在UI中显示pipeline的动态流程图)

    Html publisher (常用于发布unit test后的html report,这些html文件其实是被相关test生成好的, publisher then renders it)

    1
    2
    3
    4
    5
    6
    7
    publishHTML([allowMissing: true,
    alwaysLinkToLastBuild: true,
    keepAll: true,
    reportDir: "$WORKSPACE/cognitive-designer-api/DSJsonApiServletJUnitTests/build/reports/tests/payloadtests",
    reportFiles: 'index.html',
    reportName: 'Payload Test',
    reportTitles: ''])

    Green Balls (show green color for success)

    Blue Ocean (embedded site with new Jenkins UI)

    Job DSL: Allowing jobs to be defined in a programmatic form in a human readable file.

    Pipeline compatible plugins: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md

    在初始化设置Jenkins的时候,有可能有plugins安装失败,可以自己在 Manage plugin 中安装,然后restart Jenkins (关于restart Jenkins请在没有job运行的情况下进行,不同的安装方式restart的方法不同,或者在安装plugin的时候选择restart jenkins after install), for example: systemctl restart jenkins , 这可以消除控制面板上的plugin failure警告。。

    Continuous delivery

    In Blue Ocean , you can run multiple builds in parallel. if more than one builds run in the same agent, the workplace path is distinguished by suffix (@count number). 但是能不能run multiple builds in one agent depends on how you design your pipeline and tasks.

    Blue Ocean 中的UI对parallel的显示也很直观,方便查看。

    Trigger builds remotely

    Set pipeline can be triggered remotely by URL, also optionally set pipeline trigger token in pipeline configure UI (can be empty).

    You also need to know the user token, set from current user profile menu, you must keep the your user token to somewhere, for example, store it in credential secret text. So you can refer the token for example:

    1
    2
    TRIGGER_USER = "chengdol.example.com"
    TRIGGER_USER_TOEKN = credentials('<token id>')

    Then in the upstream pipeline script, trigger other pipeline by running curl command:

    1
    2
    3
    4
    ## can be http or https connection
    ## --user is the jenkin user and its token or password
    ## token=${PIPELINE_TRIGGER_TOKEN} can be ignored if it's empty
    curl --user ${TRIGGER_USER}:${TRIGGER_USER_TOEKN} --request POST http/https://<url>/job/${PIPELINE_NAME}/buildWithParameters?token=${PIPELINE_TRIGGER_TOKEN}\\&para1=val1\\&&para2=val2

    You don’t need to specify all parameters in URL, the parameters default values will be used if they are not specified in the URL.

    Notice that para1 and para2 must exist in parameters section of the triggered pipeline, otherwise you cannot use them. So far, based on testing, I can pass string, bool and file parameter types.

    Check Status of Another pipeline

    reference: https://gist.github.com/paul-butcher/dc68adc1c05ca970158c18206756dab1

    1
    curl --user ${LOGIN_USER}:${LOGIN_USER_TOEKN} --request GET http/https://<url>/job/${PIPELINE_NAME}/<build number>/api/json

    Then you can parse the json returned: artifacts -> result: SUCCESS, FAILURE, ABORTED

    Flyweight executor

    Flyweight executor reside in Jenkins master, used to execute code outside of node allocation. Others are heavyweight executors. Flyweight executor will not be counted into executor capacity.

    For example, we use flyweight executor for pause, in Jenkins script: https://stackoverflow.com/questions/44737036/jenkins-pipeline-with-code-pause-for-input

    1
    2
    ## see below declarative pipeline demo code
    input "waiting for approval, move to staging stage..."

    The job will be paused, you can go to Paused for Input to decide what to do next: proceed or abort. (In Blue Ocean , the pause interface is more clear)

    Declarative pipeline

    Please use Groovy style to wirte declarative pipeline!

    github demo: https://github.com/sixeyed/jenkins-pipeline-demos

    Declarative Pipelines: https://www.jenkins.io/doc/book/pipeline/syntax/

    这里有关于declarative pipeline的介绍视频, Jenkins file lives in source control! https://www.jenkins.io/solutions/pipeline/ ,

    Using Blue Ocean to setup pipeline from github need personal access token (must check out repo and user options): https://www.youtube.com/watch?v=FhDomw6BaHU

    In Jenkins UI, go to pipeline syntax then declarative directive generator , it will help you generate pipeline code for declarative pipeline: https://www.jenkins.io/doc/book/pipeline/getting-started/#directive-generator

    Jenkins parameters variables can be accessed by both params and env prefix, see this issue .

    This is just a basic structure demo:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    pipeline {
    // default agent specify
    agent any
    // pipeline level env var, global scope
    environment {
    // you can put release number here
    // referred by env.RELEASE
    RELEASE = '1.1.3'
    }
    // can have multiple stages
    stages {
    // list tool version
    stage('Audit tools') {
    steps {
    sh '''
    git version
    docker version
    '''
    }
    }
    stage('Build') {
    // agent specify
    agent any
    // stage level env var, stage scope
    environment {
    USER = 'root'
    }
    steps {
    echo "this is Build stage"
    // executable in your repo
    sh 'chmod +x ./build.sh'
    // 把jenkins中一个名为api-key的密匙的值 放入 API_KEY这个环境变量中
    // 且这个API_KEY仅在block中可见
    withCredentials([string(credentialsId: 'api-key', variable: 'API_KEY')]) {
    sh '''
    ./build.sh
    '''
    }
    }
    }
    // can have different type
    stage('Test') {
    environment {
    LOG_LEVEL = "INFO"
    }
    // parallel tasks
    parallel {
    // they can be running on different agent
    // depends on you agent setting
    stage('test1')
    {
    steps {
    // show current stage name test1
    echo "parallel ${STAGE_NAME}"
    // switch to ./src directory
    dir('./gradle') {
    sh '''
    ./gradlew -p xxx test1
    '''
    }
    }
    }
    stage('test2')
    {
    steps {
    echo "parallel ${STAGE_NAME}"
    }
    }
    stage('test3')
    {
    steps {
    echo "parallel ${STAGE_NAME}"
    }
    }
    }
    }
    stage('Deploy') {
    // waiting for user input before deploying
    input {
    message "Continue Deploy?"
    ok "Do it!"
    parameters {
    string(name: 'TARGET', defaultValue: 'PROD', description: 'target environment')
    }
    }
    steps {
    echo "this is Deploy with ${env.RELEASE}"
    // groovy code block
    // potential security hole, jenkins will not make it easy for you
    script {
    // you need to approve use of these class/method
    if (Math.random() > 0.5) {
    throw new Exception()
    }
    // you can use try/catch block for security reason
    }
    // if fail, this wouldn't get executed
    // write 'passed' into file test-results.txt
    writeFile file: 'test-results.txt', text: 'passed'
    }
    }
    }

    post {
    // will always be executed
    always {
    echo "prints whether deploy happened or not, success or failure."
    }
    // others like: success, failure, cleanup, etc
    success {
    // archive files
    archiveArtifacts 'test-results.txt'
    // slack notifation
    slackSend channel: '#chengdol-private',
    message: "Release ${env.RELEASE}, success: ${currentBuild.fullDisplayName}."
    }
    failure {
    slackSend channel: '#chengdol-private',
    color: 'danger',
    message: "Release ${env.RELEASE}, FAILED: ${currentBuild.fullDisplayName}."
    }
    }
    }

    if you don’t want to checkout SCM in stages that run in same agent, you can use this option:

    1
    2
    3
    options {
    skipDefaultCheckout true
    }

    Configure slack

    This is a slightly out-of-date video, 其实在各自pipeline中可以自己单独配置token以及选择channel。 https://www.youtube.com/watch?v=TWwvxn2-J7E

    First install slack notifaction plugin. Then go to Manage Jenkins -> Configure System , scroll down to bottom you will see slack section, see question mark for explanation.

    Then go to your target slack channel, select Add an app , search Jenkins CI , then add it to slack, follow the instructions to get the secret token, add this token to Jenkins credentials and use it in above slack configuration.

    After all set, try Test connection , you will see message in your slack channel.

    Reusable

    Reusable functions and libraries are written in Groovy . https://www.eficode.com/blog/jenkins-groovy-tutorial

    Let’s see some demos:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    pipeline {
    agent any
    // options
    parameters {
    booleanParam(name: 'RC', defaultValue: false, description: 'Is this a Release Candidate?')
    }
    environment {
    VERSION = "0.1.0"
    VERSION_RC = "rc.2"
    }
    stages {
    stage('Audit tools') {
    steps {
    // call function
    auditTools()
    }
    }
    stage('Build') {
    environment {
    // call function
    VERSION_SUFFIX = getVersionSuffix()
    }
    steps {
    echo "Building version: ${VERSION} with suffix: ${VERSION_SUFFIX}"
    sh 'dotnet build -p:VersionPrefix="${VERSION}" --version-suffix "${VERSION_SUFFIX}" ./m3/src/Pi.Web/Pi.Web.csproj'
    }
    }
    stage('Unit Test') {
    steps {
    // switch directory
    dir('./m3/src') {
    sh '''
    dotnet test --logger "trx;LogFileName=Pi.Math.trx" Pi.Math.Tests/Pi.Math.Tests.csproj
    dotnet test --logger "trx;LogFileName=Pi.Runtime.trx" Pi.Runtime.Tests/Pi.Runtime.Tests.csproj
    '''
    mstest testResultsFile:"**/*.trx", keepLongStdio: true
    }
    }
    }
    stage('Smoke Test') {
    steps {
    sh 'dotnet ./m3/src/Pi.Web/bin/Debug/netcoreapp3.1/Pi.Web.dll'
    }
    }
    stage('Publish') {
    // condition
    when {
    expression { return params.RC }
    }
    steps {
    sh 'dotnet publish -p:VersionPrefix="${VERSION}" --version-suffix "${VERSION_RC}" ./m3/src/Pi.Web/Pi.Web.csproj -o ./out'
    archiveArtifacts('out/')
    }
    }
    }
    }

    // groovy methods, can run straight groovy code
    def String getVersionSuffix() {
    if (params.RC) {
    return env.VERSION_RC
    } else {
    return env.VERSION_RC + '+ci.' + env.BUILD_NUMBER
    }
    }

    def void auditTools() {
    sh '''
    git version
    docker version
    dotnet --list-sdks
    dotnet --list-runtimes
    '''
    }

    Shared library

    Demo structure and code: https://github.com/sixeyed/jenkins-pipeline-demo-library Invoking shared library at head of jenkins file:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    // this is dynamic reference, explicitly specify the library in jenkins file
    library identifier: 'jenkins-pipeline-demo-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
    remote: 'https://github.com/sixeyed/jenkins-pipeline-demo-library.git',
    // if the repo is private, you can have credential here
    credentialsId: '<credential id>'])

    pipeline {
    agent any
    stages {
    stage('Audit tools') {
    environment {
    // pass parameters as map
    VERSION_SUFFIX = getVersionSuffix rcNumber: env.VERSION_RC, isReleaseCandidate: params.RC
    }
    steps {
    auditTools()
    }
    }
    }
    }

    You can add Global Pipeline Libraries in Configure Jenkins for any pipeline use. 这种情况下,可以设置默认的shared library,然后在jenkins file中直接调用相关函数。

    Shared pipelines

    You can put the shared pipeline into a shared library, for example:

    1
    2
    3
    4
    5
    6
    library identifier: 'jenkins-pipeline-demo-library@master', 
    retriever: modernSCM([$class: 'GitSCMSource', remote: 'https://github.com/sixeyed/jenkins-pipeline-demo-library.git'])

    crossPlatformBuild repoName: 'sixeyed/pi-psod-pipelines',
    linuxContext: 'm4',
    windowsContext: 'm4'

    Shared library is under vars folder. In Groovy, we can add a method named call to a class and then invoke the method without using the name call , crossPlatformBuild is actually the file name, inside file there is a call method.

    Multi-branch pipeline

    https://www.jenkins.io/doc/book/pipeline/multibranch/ Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.

    Orphaned item strategy, for deleted branch, you can discard or reserve it.

    Pipeline development tools

    Validating pipeline syntax

    First enable anonymous read access in Configure Global Security : https://www.jenkins.io/doc/book/pipeline/development/#linter

    Issue curl command:

    1
    2
    ## if not success, it will show you the overall problems with your jenkins file
    curl -X POST -F "jenkinsfile=<[jenkins file path]" http://<IP>:8080/pipeline-model-converter/validate

    Visual Studio code has Jenkins linter plugin, you need to configure it with linter url.

    Restart or replay

    In every build interface, restart from stage , you can select which stage to restart (sometimes stage may fail due to external reason), replay , you can edit your jenkins file and library then rerun, the changes only live in current build (after succeed, check in your updates to source control).

    Unit test

    https://github.com/jenkinsci/JenkinsPipelineUnit

  • Supports running pipelines and library methods
  • Can mock steps and validate calls
  • Jenkins with Docker

    学习步骤: 首先是agent 使用docker,然后master + agent 都使用docker, 最后交由K8s去管理。

    这块很有意思,加入容器后就太灵活了,只要有build agent支持docker且已安装,则jenkins就可以把container运行在之上。 https://www.jenkins.io/doc/book/pipeline/docker/

    https://hub.docker.com/r/jenkins/jenkins you can parallelly run several jenkins version in one machine, for purposes like testing new features, testing upgrade, so on and so forth. But you may need to customize the jenkins docker image and expose to different port.

  • containers as build agents
  • customizing the build container
  • using the docker pipeline plugin
  • Jenkins master and slave with docker: https://medium.com/@prashant.vats/jenkins-master-and-slave-with-docker-b993dd031cbd

    From agent syntax: https://www.jenkins.io/doc/book/pipeline/syntax/#agent

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    agent {
    docker {
    image 'myregistry.com/node'
    // can pass argument to docker run
    args '-v /tmp:/tmp'
    // the node must pre-configured to have docker
    label 'my-defined-label'
    // optional set the registry to pull image
    registryUrl 'https://myregistry.com/'
    registryCredentialsId 'myPredefinedCredentialsInJenkins'
    }
    }

    If there is no label option, Jenkins will dynamically provisioned on a node and it will fail if no docker installed, you can set docker label to filter: https://www.jenkins.io/doc/book/pipeline/docker/#specifying-a-docker-label

    I got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: https://stackoverflow.com/questions/47854463/docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socke

    还解决了一个权限的问题,在实验中container默认用户是 jenkins ,没有root权限无线运行container内部的程序,解决办法是 args '-u root' 在上面的配置中。

    此外jenkins会把workspace注入到container中,通过环境变量查找:

    1
    2
    sh 'printenv'
    sh 'ls -l "$WORKSPACE"

    Additionally, you can use agent with Dockerfile and install Docker Pipeline plugin.

     
    推荐文章