Skip to content
/ devops-coord-framework Public

Open-source DevOps framework example for on-boarding 1:N projects Jenkins (or other CI tool) & CLI use

License

MIT license
9 stars 8 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

tpayne/devops-coord-framework

BranchesTags

Repository files navigation

Devops Coordination Framework

If you find this framework useful or would like something added to it, feel free to drop me a comment.

Status

Framework Status: Ready for use

Build CI/Testing Status

The following indicates the CI and coverage status. SonarCloud analysis is in prototyping.

Java CI with Maven CodeQL pages-build-deployment SonarCloud Code Analysis

Contents

Overview

This repository delivers a framework that was created to help manage the DevOps process for releases that involve a large number of components and/or that need a standard process for managing components and how they are delivered.

The framework provides a set of objects intended to manage each major phase of a CI/CD pipeline into which custom logic can be placed that will then be execute at the correct place, so removing the need for linking jobs together or building custom flow logic into your pipeline.

For example, a Build object is present that allows you to register callbacks for operations like: -

  • Cleaning work areas
  • Getting code
  • Running a build process
  • Running a unit-test process
  • Running a static code analysis process etc.

All you need to do is register callbacks to do your own specific build logic or unit-test process etc. and the framework will manage all the running of that process and the coordination with other steps.

The framework also provides a small set of "most commonly used functions" service library to help "kick-start" your CI/CD process if needed. These functions include operations like: -

  • Slack notification
  • Email
  • Container services
  • SCM services
  • various file based services

These can be used (if needed) from Jenkins or command line.

Framework Plugins for Jenkins

The framework provides two different plugins for Jenkins. The first of these is a shared library which is used to provide the CI/CD framework (and service) objects discussed below. The second is a set of DSL extensions which - if installed as a Jenkins plugin - provides a set of DSL extensions which can be used in a Jenkins pipeline syntax. These are also discussed below.

Framework Objects

The framework is split into two main pipelines: -

  • The CI process for individual products/components
  • The CD process for integrated releases

The CI process is used to manage the build, deploy and test processes for each individual component and the CD process is used to manage the integration and release phases.

The main interface objects for controlling these phases are shown below: -

Class Description
CIFramework Which is used to control your CI process (Build, Deploy and Test process)
CDFramework Which is used to control your CD process (Integration and Release candidate verification process)

The CIFramework interface is split into three main phases - each of which are controlled by a specific object: -

Class Description
Build Which is used to control your build process
Deploy Which is used to control your deployment process
Test Which is used to control your test process

Likewise, the CDFramework interface is split into two main phases - each of which are controlled by a specific object as well: -

Class Description
Integration Which is used to control your integration process
ReleaseCandidate Which is used to control your Release candidate process

These interface objects allow you to register the callbacks that you want to use for each main process - like build - and then the framework will take care of running these callbacks in the correct flow and managing the coordination between the steps.

Service Objects

The framework also provides a number of "service" objects aimed at providing commonly required DevOps capabilities like cloning code, running Docker images, sending emails etc. These services are provided to help make a DevOps implementation easier to do by providing working services and utilities that are commonly required.

Class Description
Notifications For email and Slack IRC messaging
SCM For GIT and SVN SCM function(s) - currently only supporting cloning
Container For various container management commands - currently only supporting Docker
Repository For pushing and pulling files from repos - currently only file, Artifactory and Nexus OSS v2 are supported
ComponentManifest For maintaining your manifest of integrated components. This is the list of component names, versions, status and locations that you register with the manifest. These can then be accessed later on for usage in other testing or release processes
Provision For running provisioning scripts - currently only Ansible is supported

So, why have a DevOps framework

As mentioned above, the framework is primarily provided as a way of allowing you to control - in a standard way - how products are on-boarded into CI/CD and how that process is then managed. It does this in two ways. Firstly, by providing a standard framework into which product-teams can implement their CI processes, so ensuring everyone is using the same guidelines. Secondly, by having the framework manage the process flow of the overall steps and the coordination between them.

The Problem Statement

As an organization with many products or different components in place, transferring these all over to a CI/CD process may be a bit daunting - especially if they each use different types of technologies or build processes. Without a framework in place, those products may each end up implementing their own CI/CD process or technology, making the management and integration of those products a big challenge.

Integration of products and the management of that process is also a big problematic area. A release that is made up of many products means that you need to track which versions of products have been released, which versions are in test and how they can work with each other. If, for example, your release if made up of 20 different products managing the status and dependencies of those products could be a large task. This is something the framework can do for you by managing the list of components and how they are promoted through the release pipeline.

Also, by extending the base framework, you can implement specific customisations that you might want to apply to every product - like security scans - that cannot be "forgotten" or missed when teams implement their own CD processes.

The Implementation

The framework has been implemented using Groovy/Java based callbacks. This means that you can easily integrate it into Jenkins pipelines as a shared library or use it standalone. The callback mechanism means that virtually any CLI or API based build technology can be run using it, so it is totally flexible regarding the CI/CD processes it can wrap.

Examples of both standalone and Jenkins pipeline implementations are provided.

Framework Process Flows

The process flows provided by the framework are described below.

The Overall Process Flow

The overall process flow is that used to implement the CI/CD pipeline. The picture below shows how this works.

Overall Process flow

Individual products use a standard CI build, deploy and test process to verify their changes are working as expected. These are then promoted to the component manifest for further testing in an integrated flow.

In the context of a release end to end flow, the framework works as shown in the picture below. The CI process is used for feature development and the CD process is used for the release verification. Quality gates are implemented using the test processes to determine if components should be added to the component manifest and to determine if the release candidate can be promoted to production.

E2E Process flow

The CI Process Flow

The CI process flow controls the component or product-level build, deploy and test process. This pipeline works as follows.

CI Process flow

When a product has finished with the testing process, then it will be promoted to the component manifest to make it a candidate for integration testing.

The CD Process Flow

When an update is detected to the component manifest, this kicks off the CD flow which then runs the integration and other release verification processes to ensure that the release stack is ready for deployment to production. This pipeline flow is shown below.

CD Process flow

Jenkins and Compiler Support

The framework will only run with the following: -

  • JDK 8+ (javac 1.8.0_201+)
  • Jenkis LTS 2.164.1+

You can downgrade the versions by modifying the pom.xml file, but you will also need to downgrade any dependent plugins.

The framework has only been compiled and tested using the documented versions above (and the latest versions of Jenkins 2.168).

Framework Classes

The following are the main framework classes and the methods that they have.

Build Class

The Build class is provided to control your build process and has the following methods: -

Method Description
prepareWorkArea() A callout provided to help prepare your workarea for a build
getCode() A callout provided to help you pull your code
preBuild() A callout provided to help prepare for a build
runBuild() A callout provided to help run a build process
postBuild() A callout provided to help run a post build process
runUnitTests() A callout provided to help run any unit tests
evaluateUnitTests() A callout provided to help evaluate any unit tests results
runStaticCodeTests() A callout provided to help run any static code analysis process
evaluateStaticCodeTests() A callout provided to help evaluate any analysis results
bakeImage() A callout provided to help bake an image
uploadAssets() A callout provided to help upload any assets created during the build
logResults() A callout provided to help log any results
promote() A callout provided to help promote a build to the next phase
runPipeline() Run the pipeline

All callbacks are run in the above order, no matter how your register them.

Deploy Class

The Deploy class is provided to control your deploy process and has the following methods: -

Method Description
prepareForDeploy() A callout provided to help prepare your environment for a deploy
getAssets() A callout provided to help you pull your deploy assets
preDeploy() A callout provided to help prepare for a deploy
runDeploy() A callout provided to help run a deploy process
postDeploy() A callout provided to help run a post deploy process
runSmokeTests() A callout provided to help run any smoke tests
evaluateSmokeTests() A callout provided to help evaluate any smoke tests results
logResults() A callout provided to help log any results
promote() A callout provided to help promote a deploy to the next phase
runPipeline() Run the pipeline

All callbacks are run in the above order, no matter how your register them.

Test Class

The Test class is provided to control your test process and has the following methods: -

Method Description
prepareForTest() A callout provided to help prepare your environment for testing
getAssets() A callout provided to help you pull your test assets
preTest() A callout provided to help prepare for a test
runTest() A callout provided to help run a test process
postTest() A callout provided to help run a post test process
evaluateTests() A callout provided to help evaluate any tests results
logResults() A callout provided to help log any results
promote() A callout provided to help promote a test to the next phase
runPipeline() Run the pipeline

All callbacks are run in the above order, no matter how your register them.

Integration Class

The Integration class is provided to control your integration process and has the following methods: -

Method Description
getComponentList() A callout provided to help get the component list for processing
prepareForDeploy() A callout provided to help prepare an environment for use
getDeployAssets() A callout provided to help pull your deployment assets
preDeploy() A callout provided to help prepare for the deploy
runDeploy() A callout provided to help run the deploy
postDeploy() A callout provided to help perform any post deployment actions
runSmokeTests() A callout provided to help run smoke tests
evaluateSmokeTests() A callout provided to help evaluate the smoke test results
logDeployResults() A callout provided to help log the deployment results
prepareForTest() A callout provided to help prepare for testing
getTestAssets() A callout provided to help pull test assets
preTest() A callout provided to help prepare for the testing
runTests() A callout provided to help run the testing
postTest() A callout provided to help perform any post testing activities
evaluateTestResults() A callout provided to help evaluate test results
logTestResults() A callout provided to help log test results
promote() A callout provided to help promote a test to the next phase
runPipeline() Run the pipeline

All callbacks are run in the above order, no matter how your register them.

ReleaseCandidate Class

The ReleaseCandidate class is provided to control your release process and has the following methods: -

Method Description
getComponentList() A callout provided to help get the component list for processing
prepareForDeploy() A callout provided to help prepare an environment for use
getDeployAssets() A callout provided to help pull your deployment assets
preDeploy() A callout provided to help prepare for the deploy
runDeploy() A callout provided to help run the deploy
postDeploy() A callout provided to help perform any post deployment actions
runSmokeTests() A callout provided to help run smoke tests
evaluateSmokeTests() A callout provided to help evaluate the smoke test results
logDeployResults() A callout provided to help log the deployment results
prepareForTest() A callout provided to help prepare for testing
getTestAssets() A callout provided to help pull test assets
preTest() A callout provided to help prepare for the testing
runTests() A callout provided to help run the testing
postTest() A callout provided to help perform any post testing activities
evaluateTestResults() A callout provided to help evaluate test results
logTestResults() A callout provided to help log test results
rollback() A callout provided to help perform a rollback if required
finish() A callout provided to help perform any final actions if needed
runPipeline() Run the pipeline

All callbacks are run in the above order, no matter how your register them.

In addition, every callback takes the following parameters: -

Parameter Description
body:{} Used to specify the Groovy code to run the process
finalHandler:{} Used to specify any Groovy code which will be invoked after the process has run
exceptionHandler:{} Used to specify any Groovy code which will be invoked if any exception occurs

CIFramework Class

The CIFramework class is provided to control your CI process and has the following methods: -

Method Description
setBuild() A callout provided to set your build object into the CI framework for processing
getBuild() A callout provided to get your build object from the CI framework
setDeploy() A callout provided to set your deploy object into the CI framework for processing
getDeploy() A callout provided to get your deploy object from the CI framework
setTest() A callout provided to set your test object into the CI framework for processing
getTest() A callout provided to get your test object from the CI framework
launchCI() A callout provided to run the CI process

CDFramework Class

The CDFramework class is provided to control your CD process and has the following methods: -

Method Description
setIntegration() A callout provided to set your integration object into the CD framework for processing
getIntegration() A callout provided to get your integration object from the CD framework
setReleaseCandidate() A callout provided to set your release candidate object into the CD framework for processing
getReleaseCandidate() A callout provided to get your release candidate object from the CD framework
launchCD() A callout provided to run the CD process

How to Install

To install this Jenkins share library, do the following...

1) git clone https://github.com/tpayne/devops-framework.git
2) cd devops-framework
3) mvn package
4) cd devops-framework-pipeline/target/
5) Unzip devops-framework-pipeline-dsl-pack.zip into a working directory
6) Use the instructions in the Jenkins Wiki (https://jenkins.io/doc/book/pipeline/shared-libraries/#global-shared-libraries) to install the shared library into your Jenkins system

You will need to configure the unit-tests as discussed below and install/configure Docker and Ansible as they are used during the test process. If you want to just build the packages without doing any unit-tests, then you can do this via...

mvn clean package -Dmaven.test.skip=true

This will build the packages only.

The Jenkins DSL HPI plugin will automatically be built via the mvn package command. The HPI file is situated in the devops-framework-plugin/target directory and is loaded into Jenkins using the advanced plugin manager option in the Jenkins management console. This is the same as any standard file-based Jenkins plugin.

Service Classes

The following are the main service classes and the methods that they have.

Utilities

This class provides various useful utilities that are used and has the following methods: -

Method Description
isUnix() Used to detect UNIX based OS
isWindows() Used to detect Windows based OS
mapProperties() Used to map a properties file into a Map
getDefaultProperties() Used to read any default properties that might have been setup for the framework to use
readAllBytes() Used to read a file into memory as an array of bytes. Mostly for binary files
readFile() Used to read a text file into memory as a string
writeFile() Used to write strings or bytes to a file
getExecutable() Used to locate an executable file in the path and return a File object to it
runCmd() Used to run a shell command and trap any output if wanted
getTmpDir() Used to return a File object to the temporary directory setup on the machine
deleteDirs() Used to emulate rm -fr
copyFile() Used to copy files
copyDirectories() Used to copy directories
getFileExt() Used to get a file extension
calcFileMD5() Used to get a calculate a file's MD5 checksum

SCM

This class provides all the SCM related support and has the following methods: -

Method Description
scmClone() Used for cloning code from supported SCM repos like Git or SVN

Notifications

This class provides notification related functionality and has the following methods: -

Method Description
sendMail() Used for sending email text or HTML/text notifications
messageSlackChannel() Used for sending notifications to Slack

Container

This class provides container related functionality and has the following methods: -

Method Description
createContainerRegistry() Used to create a container register for supported platforms like Docker
pushContainer() Used to push an image to a container register
tagContainer() Used to tag a container for future use
deleteContainerRegistry() Used to delete a container register
pullContainerImage() Used to pull a container image from somewhere
deleteContainerImage() Used to delete a container image
runContainer() Used to run a container
buildContainer() Used to build or bake a container from a given build file

Repository

This class provides repository related functionality and has the following methods: -

Method Description
pullAssetFromRepo() Used to pull an asset from a repo
pushAssetToRepo() Used to push an asset to a repo

ComponentManifest

This class provides repository related functionality for maintaining your component manifest list (used in releases) and has the following methods: -

Method Description
isValid() Used to check the manifest is valid
getRepo() Used to get the repo file details
setRepo() Used to set the repo file details
getCommitComment() Used to get the last commit comment
getCommitter() Used to get the last committer
getCommitDate() Used to get the last commit date
getManifestVersion() Used to get the manifest version
setManifestVersion() Used to set the manifest version
getManifestStatus() Used to get the manifest status
setManifestStatus() Used to set the manifest status
getComponentList() Used to get the list of components registered in the manifest
getComponent() Used to get the details for an individual component
addComponent() Used to add a component to the manifest
updateComponent() Used to update the details for an individual component registered in the manifest
removeComponent() Used to remove a component from the manifest
convertManifestToJSON() Used to convert the component manifest to a JSON string
commit() Used to commit the manifest details to the repo file

The manifest file (by default) is a JSON file stored in the repository file. This manifest file looks like this: -

{
  "commitComment": "Committed version 30 by user1 at 13:20:12 27/03/2019",
  "compList": {
    "gitplugin": {
      "componentVersion": "1553692812066",
      "componentName": "gitplugin",
      "componentLocation": "/Volumes/WorkDisk/tmp/Repos/github.hpi.1553692812066",
      "componentStatus": "Integration Test",
      "componentMd5Sum": "9f8628f68ce0865348ade6d2a4d568af"	      
    },
    "dimensionsscm": {
      "componentVersion": "1553692305487",
      "componentName": "dimensionsscm",
      "componentLocation": "/Volumes/WorkDisk/tmp/Repos/dimensionsscm.hpi.1553692305487",
      "componentStatus": "Integration Test",
      "componentMd5Sum": "7251e4a7b0d77264940db7baa8c58756"	      
    },
    "JavaAppCICD": {
      "componentVersion": "1553609424273",
      "componentName": "JavaAppCICD",
      "componentLocation": "/Volumes/WorkDisk/tmp/Repos/jpetstore.war.1553609424273",
      "componentStatus": "Integration Test",
      "componentMd5Sum": "db8c6088e5e789db8c850075d98dc373"	      
    }
  },
  "committer": "user1",
  "status": "Integration Test",
  "version": "30",
  "commitUTCDate": 1553692812367
}

There are a number of parts to it as follows: -

Key Description
commitComment Used to hold the last commit comment
compList Used to hold the registered components
committer Used to register who did the last commit
status Used to hold the current status
version Used to hold the current version
commitUTCDate Used to hold the last commit date/time in UTC format

The compList is formatted as: -

Key Description
<componentKey> Used to hold the component key, e.g. JavaAppCICD
componentVersion Used to hold the registered component version
componentName Used to hold the registered component name
componentLocation Used to hold the registered component location, i.e. where it is stored
componentStatus Used to hold the registered component status
componentMd5Sum Used to hold the registered component MD5 checksum

The values of these keys are free text and are set by the appropriate set...() methods. What formats you use are up to you.

Further keys can be added by modifying the ComponentManifest.groovy file as needed.

Provision

This class provides provisioning related functionality and has the following methods: -

Method Description
runPlaybook() Used to run a play/runbook (Ansible only)

How to Use

The shared library works by providing controller classes that you can use in a pipeline job. As such, you need to create a pipeline job and register the callbacks that you want to use.

For example, a sample pipeline might look like

// Sample pipeline...
@Library('devops-framework')
import org.devops.*

// Include the core services as I want to use them as well...
@GrabResolver(name='devops-core', root='file:///Volumes/WorkDisk/GROOVY/devops-framework/devops-framework-core/target/lib/')
@Grab('org.devops.framework.core:devops-framework-core:0.0.1-SNAPSHOT')

// Import the framework core classes...
import org.devops.framework.core.*

def config = [
    property1: 'value1',
    property2: 'value2',
    property3: 'value3'
]

def bld = new org.devops.Build(this, config)

node {    
        File fetchDir = new File("/Volumes/WorkDisk/tmp/BuildJobs/")
	String scmURI = "https://github.com/sourcerepo/JavaAppCICD.git"
	String slackURI = "https://hooks.slack.com/services/SDJSHETEJDKRJFHFIDLODJFF"

	// Register my build process...    
	bld.runBuild(body:{
	    // This is a Maven which will compile and run all the unit tests as
	    // part of the process, so we do not need all the other build callbacks...
	    println ">Run build...<"
	    sh(script: "cd ${fetchDir.getAbsolutePath()}; chmod +rx ./mvnw; ./mvnw -q package")
	}, finalHandler:{ println ">Build Job Done<" })

	// Register my get code callback...
	bld.getCode(body:{
	    println ">Get code - cloning from GIT...<"
	    StringBuffer outputStr = new StringBuffer()
	    boolean retStat = SCM.scmClone(ConfigPropertiesConstants.SCMGIT,
					    scmURI,fetchDir,outputStr)
	    if (retStat) {
		println "Code clone worked"
		Notifications.messageSlackChannel(slackURI,"${label}: Clone worked")
	    } else {
		println "Code clone failed"
		println outputStr.toString()
		Notifications.messageSlackChannel(slackURI,"${label}: Code clone failed - "+outputStr.toString())
	    }  
	    outputStr = null
	}) 

	// Register my bake callback...
	bld.bakeImage(body:{ println ">Run my bake<" })

	// Register a prepareWorkArea callback...
	bld.prepareWorkArea(body:{
	    println ">Prepare Work Area - clean my files up...<"
	    if (fetchDir.exists()) {
		Utilities.deleteDirs(fetchDir)
	    }
	    fetchDir.mkdirs()
	})

	// Run my pipeline...
	bld.runPipeline()
}

This will run the prepareWorkArea(), getCode(), runBuild() and bakeImage() callbacks in this order.

A DSL plugin example would look like this

// Sample DSL plugin calls...
node {

    // Clone code...
    stage('clone') {
	// Some constants...
	String fetchDirG = "/Volumes/WorkDisk/tmp/BuildJobs/git/"
	String fetchDirS = "/Volumes/WorkDisk/tmp/BuildJobs/svn/"
	String scmURI = "https://github.com/jenkinsci/dimensionsscm-plugin.git"
	// Which host is the script running on?
	sh label: '', script: 'uname -a; hostname'

	def status = devOpsFrameworkGitCloneStep repoName: scmURI,
		 targetDir: fetchDirG
	println status

	status = devOpsFrameworkSvnCloneStep repoName: scmURI,
		 targetDir: fetchDirS
	println status
    }

    // Container ops with Docker...
    stage('container-ops') {

	def status = devOpsFrameworkPullContainerStep containerName: 'tomcat'
	println status

	status = devOpsFrameworkRunContainerStep cmdStr: 'ls -l /',
			containerName: 'tomcat'
	println status

	status = devOpsFrameworkPushContainerStep imageName: 'pushImage'
	println status

	status = devOpsFrameworkTagContainerStep containerName: 'tomcat',
				targetName: 'yumi-target'
	println status

	status = devOpsFrameworkRmContainerStep containerName: 'tomcat', force: true
	println status

	status = devOpsFrameworkBuildContainerStep buildDirectory: '/home/Jenkins/workspace/TestDSL/',
			containerFile: '/home/Jenkins/workspace/TestDSL/DockerFileSuse.test',
			containerName: 'dsfsdf'
	println status

	// Ansible - not container op...
	status = devOpsFrameworkAnsibleRunbookStep hostFile: '/etc/ansible_hosts',
		    runFile: '/tmp/playbook.yml',
		    workingDir: '/tmp'
	println status
    }
}

Framework Documentation

The framework code comes with some limited JavaDoc present which can be used to generate API documentation by using the command

% mvn gplus:generateStubs gplus:groovydoc

This will generate JavaDoc style comments in the target/ directory

Class Usage Examples and Running Unit-tests

Virtually all the class functions documented above have example unit/functional tests which are run in the Maven test phase.

You can use these as examples to show you how to use the classes.

Running the unit-tests

The values used in the unit-tests for things like slack channels and GitHub repos are currently hardcoded to working values. However, these will need to be changed if you wish to run the tests. These values are located in a properties file unitTest.properties.

These will need to be modified to values more specific for you otherwise some of the tests may fail.

(Note - The tests have very little documentation imbedded in them currently. This will be added slowly).

Pipeline Examples

Examples of pipelines created using the framework can be found in the examples directory.

Example Description
buildJenkinsPluginPipeline.txt Example build pipeline that fetches code, builds it, commits it to a repo, then updates a component manifest with the new version
buildJenkinsPluginPipelineWithSlackNotif.txt As buildJenkinsPluginPipeline.txt, but also includes Slack channel notifications and shows how the component manifest can track many different components
CIJenkinsPluginPipeline.txt Example build pipeline that fetches code, builds it, commits it to a repo, creates a baked Docker image, then updates a component manifest with the new version. It is implemented using the CIFramework object
IntegrationTestPlugin.txt Example integration pipeline that fetches binaries from the component manifest, bakes a Docker image using them, then tests the container. It is implemented using the Integration object
DSLPipelineExamples.txt Example DSL using the Jenkins DSL extension plugin
ExampleComponentManifestRepo.json Example component manifest file

Screenshots

Some screenshots of the various jobs are shown below

Build test case

The following is an example run using the Build object.

Build Test case

CI Framework test case

The following is an example run using the CIFramework object.

CIFramework Test case

Integration test case

The following is an example run using the Integration object.

Integration Test case

DSL Plugin

As well as the framework objects above, the framework also provides a HPI DSL plugin which when loaded into Jenkins (as a normal plugin) adds a number of service steps which can be used in your Jenkins pipeline to get your CI/CD process off the ground. These service steps are as follows.

Step DSL Description
devOpsFrameworkBuildContainerStep This step provides a way of building container images (Docker)
devOpsFrameworkPullContainerStep This step provides a way of pulling container images (Docker)
devOpsFrameworkRunContainerStep This step provides a way of running container images (Docker)
devOpsFrameworkRmContainerStep This step provides a way of removing container images (Docker)
devOpsFrameworkTagContainerStep This step provides a way of tagging container images (Docker)
devOpsFrameworkSvnCloneStep This step provides a way of cloning a branch from a SVN repo
devOpsFrameworkGitCloneStep This step provides a way of cloning a branch from a GIT repo
devOpsFrameworkFilePushStep This step provides a way of pushing an asset to a file-based repo
devOpsFrameworkFilePullStep This step provides a way of pulling an asset to a file-based repo
devOpsFrameworkNexusPushStep This step provides a way of pushing an asset to a Nexus-based repo
devOpsFrameworkNexusPullStep This step provides a way of pulling an asset to a Nexus-based repo
devOpsFrameworkArtifactoryPushStep This step provides a way of pushing an asset to a Artifactory-based repo
devOpsFrameworkArtifactoryPullStep This step provides a way of pulling an asset to a Artifactory-based repo
devOpsFrameworkAnsibleRunbookStep This step provides a way of running an Ansible playbook
devOpsFrameworkShellCmdStep This step provides a way of running a shell command
devOpsFrameworkShellFileStep This step provides a way of running a shell script

All these plugins support master and slave configurations.

Plugin Syntax

The above steps have the following syntax: -

Building a container image

Name: devOpsFrameworkBuildContainerStep

Purpose: This step is for building a container image from a makefile

Example:

devOpsFrameworkBuildContainerStep buildDirectory: '/tmp', 
	containerFile: '/tmp/Dockerfile', containerName: 'tomcat'

Parameters:

Parameter Value Description
buildDirectory '<someDirectory>' Mandatory parameter to specify which directory to run the build in
containerFile '<fileName>' Optional parameter to specify which build file to use. The default is Dockerfile
containerName '<containerName>' Mandatory parameter to specify the container name to build
quiet true Optional parameter to suppress stdout reporting
Pulling a container image

Name: devOpsFrameworkPullContainerStep

Purpose: This step is for pulling a container image from a repo

Example:

devOpsFrameworkPullContainerStep containerName: 'tomcat'

Parameters:

Parameter Value Description
containerName '<containerName>' Mandatory parameter to specify the container name to pull
quiet true Optional parameter to suppress stdout reporting
Running a container image

Name: devOpsFrameworkRunContainerStep

Purpose: This step is for running a container image with a command

Example:

devOpsFrameworkRunContainerStep cmdStr: 'df -H', containerName: 'tomcat'

Parameters:

Parameter Value Description
containerName '<containerName>' Mandatory parameter to specify the container name to run
cmdStr '<commandString>' Mandatory parameter to specify the command string to run
quiet true Optional parameter to suppress stdout reporting
Removing a container image

Name: devOpsFrameworkRmContainerStep

Purpose: This step is for deleting a container image from the local repo

Example:

devOpsFrameworkRmContainerStep containerName: 'tomcat', force: true

Parameters:

Parameter Value Description
containerName '<containerName>' Mandatory parameter to specify the container name to delete
force '<true or false>' Optional parameter to force the container to be removed
quiet true Optional parameter to suppress stdout reporting
Tagging a container image

Name: devOpsFrameworkTagContainerStep

Purpose: This step is for tagging a container image

Example:

devOpsFrameworkTagContainerStep containerName: 'tomcat', targetName: 'taggedTomcat'

Parameters:

Parameter Value Description
containerName '<containerName>' Mandatory parameter to specify the container name to tag
targetName '<targetName>' Mandatory parameter to specify the target tag
quiet true Optional parameter to suppress stdout reporting
Clone SVN Repo

Name: devOpsFrameworkSvnCloneStep

Purpose: This step is for cloning the code from a SVN repository

Example:

devOpsFrameworkSvnCloneStep repoName: 'http://svnrepo.net/mycode/', targetDir: '/tmp/', 
	userName: 'user1', userPwd: 'user1Pwd'

Parameters:

Parameter Value Description
repoName '<repoName>' Mandatory parameter to specify the repo to clone
targetDir '<someDirectory>' Optional parameter to specify the target directory to use
userName '<userName>' Optional parameter to specify a valid SCM username
userPwd '<password>' Optional parameter to specify a valid SCM user password
Clone GIT Repo

Name: devOpsFrameworkGitCloneStep

Purpose: This step is for cloning the code from a GIT repository

Example:

devOpsFrameworkGitCloneStep repoName: 'https://github.com/myuser/myrepo.git', 
	targetDir: '/tmp/', userName: 'user1', userPwd: 'user1Pwd'

Parameters:

Parameter Value Description
repoName '<repoName>' Mandatory parameter to specify the repo to clone
targetDir '<someDirectory>' Optional parameter to specify the target directory to use
userName '<userName>' Optional parameter to specify a valid SCM username
userPwd '<password>' Optional parameter to specify a valid SCM user password
Pull File Repo

Name: devOpsFrameworkFilePullStep

Purpose: This step is for pulling files from a file-based repo

Example:

devOpsFrameworkFilePullStep srcFile: '/Volumes/FileRepo/ASD/Releases/latest/asd.hpi', 
	targetFile: '/Volumes/WorkArea/Compon_ASD/target/'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify a file or directory to pull
targetFile '<fileName>' Mandatory parameter to specify the target
Push File Repo

Name: devOpsFrameworkFilePushStep

Purpose: This step is for pushing files to a file-based repo

Example:

devOpsFrameworkFilePushStep srcFile: '/Volumes/WorkArea/Compon_ASD/target/asd.hpi', 
	targetFile: '/Volumes/FileRepo/ASD/Releases/latest/'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify a file or directory to push
targetFile '<fileName>' Mandatory parameter to specify the target
Pull Nexus Repo

Name: devOpsFrameworkNexusPullStep

Purpose: This step is for pulling files from a Nexus-based repo

Example:

devOpsFrameworkNexusPullStep srcFile: 'http://localhost:8081/nexus/sites/generic-local/comp/Asd.war', 
    targetFile: '/Volumes/WorkDisk/deploy/ASD.war', 
    userName: 'admin', userPwd: 'admin123'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify a file to pull from Nexus
targetFile '<fileName>' Mandatory parameter to specify the target
userName '<userName>' Optional parameter to specify a valid repo username
userPwd '<password>' Optional parameter to specify a valid repo user password
quiet true Optional parameter to suppress stdout reporting
Push Nexus Repo

Name: devOpsFrameworkNexusPushStep

Purpose: This step is for pushing files to a Nexus-based repo

Example:

devOpsFrameworkNexusPushStep srcFile: '/Volumes/WorkArea/Compon_ASD/target/asd.war',
    targetFile: 'http://localhost:8081/nexus/sites/generic-local/comp/Asd.war',
    userName: 'admin', userPwd: 'admin123'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify the file to push to Nexus
targetFile '<fileName>' Mandatory parameter to specify the target
userName '<userName>' Optional parameter to specify a valid repo username
userPwd '<password>' Optional parameter to specify a valid repo user password
quiet true Optional parameter to suppress stdout reporting
Pull Artifactory Repo

Name: devOpsFrameworkArtifactoryPullStep

Purpose: This step is for pulling files from a Artifactory-based repo

Example:

devOpsFrameworkArtifactoryPullStep srcFile: 'http://localhost:8081/artifactory/generic-local/comp/Asd.war', 
    targetFile: '/Volumes/WorkDisk/deploy/ASD.war', 
    userName: 'admin', userPwd: 'admin123'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify a file to pull from Artifactory
targetFile '<fileName>' Mandatory parameter to specify the target
userName '<userName>' Optional parameter to specify a valid repo username
userPwd '<password>' Optional parameter to specify a valid repo user password
quiet true Optional parameter to suppress stdout reporting
Push Artifactory Repo

Name: devOpsFrameworkArtifactoryPushStep

Purpose: This step is for pushing files to a Artifactory-based repo

Example:

devOpsFrameworkArtifactoryPushStep srcFile: '/Volumes/WorkArea/Compon_ASD/target/asd.war',
    targetFile: 'http://localhost:8081/artifactory/generic-local/comp/Asd.war',
    userName: 'admin', userPwd: 'admin123'

Parameters:

Parameter Value Description
srcFile '<fileName>' Mandatory parameter to specify the file to push to Artifactory
targetFile '<fileName>' Mandatory parameter to specify the target
userName '<userName>' Optional parameter to specify a valid repo username
userPwd '<password>' Optional parameter to specify a valid repo user password
quiet true Optional parameter to suppress stdout reporting
Run an Ansible Playbook

Name: devOpsFrameworkAnsibleRunbookStep

Purpose: This step is for running an Ansible playbook

Example:

devOpsFrameworkAnsibleRunbookStep hostFile: '/Volumes/WorkDisk/resources/ansible_hosts',
    workingDir: '/Volumes/WorkDisk/tmp/',
    runFile: '/Volumes/WorkDisk/resources/ansible_playbook.yml'

Parameters:

Parameter Value Description
hostFile '<fileName>' Mandatory parameter to specify the hosts file to use
runFile '<fileName>' Mandatory parameter to specify the playbook to use
workingDir '<dirName>' Optional parameter to specify a working directory
Run any Shell script

Name: devOpsFrameworkShellCmdStep

Purpose: This step is for running any shell script

Example:

devOpsFrameworkShellCmdStep script: '''
                    ls /tmp; echo 1; echo 2; echo 3''', 
                    workingDir: '/tmp/', quiet: true

Parameters:

Parameter Value Description
script '<shell script>' Mandatory parameter to specify shell script to run
workingDir '<dirName>' Optional parameter to specify a working directory
quiet true Optional parameter to suppress stdout reporting
Run any Shell Script File

Name: devOpsFrameworkShellFileStep

Purpose: This step is for running any shell script file

Example:

devOpsFrameworkShellFileStep cmdFile: '/users/alexgray/build.sh', 
                    workingDir: '/tmp/', quiet: true

Parameters:

Parameter Value Description
cmdFile '<fileName>' Mandatory parameter to specify shell file to run
workingDir '<dirName>' Optional parameter to specify a working directory
quiet true Optional parameter to suppress stdout reporting

Liability Warning

The contents of this repository (documents and examples) are provided “as-is” with no warrantee implied or otherwise about the accuracy or functionality of the examples.

You use them at your own risk. If anything results to your machine or environment or anything else as a result of ignoring this warning, then the fault is yours only and has nothing to do with me.

Licensing

This software is licensed using the terms and provisions of the MIT license.

Known Issues Notes

The following are known issues: -

  • The framework has not been fully ported or tested on Windows. Do not be surprised if some work needs to be done to make it work on that OS. Where possible classes have been written to support both UNIX and Windows, but they have not been tested on Windows, so some issues may occur. The framework was developed and tested on Unix.
  • The Jfrog-cli is not currently supported in the Artifactory classes. This will be added later on.
  • Currently, the framework can run both with in and without of Jenkins (if so required), but this duality is not guaranteed to be maintained in the future.
  • Container singletons have been provided to show how they can be created, but these have not been verified in usage
  • Due to a "feature" with Jenkins, any process which takes 5+ mins to run will be killed by Jenkins. This is core Jenkins pipeline and cannot be overriden without writing a custom threading plugin. The Jenkins developers seem to regard long running jobs as errors. To overcome this you will need to use the DevOps-framework DSL plugin for Jenkins. DSL functions have been added that provide the same functionality, but remove the time issue.
  • Any changes on this branch are automatically built & unit-tested for completeness using Travis-CI. If you want to view the job history, click on the 'build' status icon at the beginning on this README.
  • This LTS version is a limited implementation only. Services like K8s support and multi-cloud deployment are not available. However, feel free to implement them on top of the existing services.

About

Open-source DevOps framework example for on-boarding 1:N projects Jenkins (or other CI tool) & CLI use

Topics

devops utility-classes utility-library cicd devops-tools devops-services jenkins-pipeline devops-workflow devops-pipeline utility-function on-boarding cicd-feature cicd-jenkins-test cicd-promote-to-production cicd-jenkins

Resources

Readme

License

MIT license

Security policy

Security policy
Activity

Stars

9 stars

Watchers

4 watching

Forks

8 forks
Report repository

Releases 1

Production release Latest
May 3, 2019

Packages

 
 
 

Contributors 3

  •  
  •  
  •  

Languages