gocd-git-path-material-plugin

GoCD plugin to introduce a material that watches on a sub-directory of a Git repository.

This is the missing GoCD support for mono-repos.

What you can do with this plugin

...that you can't do with the built-in Git material:

• Support mono-repos without excessive triggering from a material that watches the entire repository
• Implement side-channels into deployment pipelines for source-controlled environment configuration
• Version control deployment configuration in the same repository as your code without rebuilding software every time unrelated configuration is changed; supporting twelve-factor apps
• Fix fan-in - materials that monitor different paths in a repo are considered separate when evaluating fan-in conditions
• Provide clean, filtered change logs for pipeline/VSM comparisons showing only changes on the monitored paths within a repository

This plugin is intended as

• a drop-in replacement for the built-in GoCD Git material (see known issues for some caveats)
• replaces the problematic use of whitelists and blacklists
  • whitelists and blacklists in GoCD tend to prevent/block fan-in because materials with different whitelists are still considered equivalent by the fan-in algorithm. This makes them unsuitable for use as environment side-channels or dealing with mono-repos.

Advanced features

Wildcard whitelists - When the Git command line is available on your agents; you can also use wildcards like config/*/prod.yaml. Anything that works with git log will work here. Shallow clones - supported in the same way as the GoCD Git Material supports them

TOC

• Installation
• Requirements
• Usage
  • Via pipelines-as-code
  • Via UI wizards
  • Via UI advanced configuration
  • Visualising changes
  • Constructing path expressions
  • Migration from v1 to v2
  • Frequently asked questions
• Contributing
  • Build
  • Run
  • Release

Installation

Manually

• Download from releases
• Follow the installation instructions here

Dockerized GoCD automatic plugin install

• Find release URL
• Set environment variable like the below; replacing ${VERSION}

  GOCD_PLUGIN_INSTALL_gocd-git-path-material-plugin=https://github.com/TWChennai/gocd-git-path-material-plugin/releases/download/v${VERSION}/gocd-git-path-material-plugin-${VERSION}.jar
  

Requirements

Plugin Version	Requirements	Recommended
2.2+	GoCD >= 19.9 Server and agent running Java 11+	GoCD 20.9 for full functionality git binary available on path
2.1	GoCD >= 18.10 Server and agent running Java 9+	At least GoCD 19.2, but >= 20.9 for full functionality git binary available on path
1.x	GoCD >= 15.1 Server and agent running Java 7+

Usage

Via pipelines-as-code

From GoCD 19.2.0 onwards you can use via pluggable material configuration with gocd-yaml-config-plugin

materials:
  path-filtered-material:
    plugin_configuration:
      id: git-path
    options:
      url: https://github.com/TWChennai/gocd-git-path-sample.git
      username: username # optional
      path: path1, path2/subpath
      shallow_clone: false # optional
    secure_options: # optional
      password: 'encrypted_value'
    destination: destDir

Via UI wizards

Available from GoCD 19.8.0 onwards.

Via UI advanced configuration

Available on all compatible versions of GoCD.

You can see a sample here.

Visualising changes

VSM view

View changes grouped by material

Constructing path expressions

This plugin relies on the underlying Git CLI to interpret path expressions. For that reason, you can construct almost any expression you could do with a git pathspec including

• including certain paths
• including only certain files deep inside a path structure
• excluding certain paths or files (using :(exclude) or :! or :^)
• using glob syntax (using :(glob))

Restrictions

• cannot contain commas (used for splitting)
• whitespace before and after splitting commas is ignored

You can normally test your expressions using something like the below to see the last revision that would have triggered a build

EXPR='path1, :(exclude)*README.md'
git log -1 $(echo "${EXPR}" | tr -d ',' )

For example, these are all valid expressions

# Monitor for changes in path1 and path2/subpath
path1, path2/subpath

# Monitor for changes in path1, but ignore README changes
path1, :(exclude)*README.md

# Monitor for changes to all config files for the qa environment, in any folder
config/*/qa.yaml

Migration from v1 to v2

v2 is a major overhaul of the plugin, and prefers the use of the git command line to the previously preferred jgit. This makes the plugin consistent with more recent GoCD server & agent Docker releases, and improves ability to support standard git features such as shallow and no-checkout clones and wildcard matches in paths.

As a result the plugin ID has changed between the two versions.

v1 plugin id	v2+ plugin id
GitPathMaterial	git-path

• If you use the gocd-yaml-config-plugin to configure your pipelines as code; you can update the ID per the above example.
• If you configure your pipelines via the UI, you will need to add new materials and remove the old ones one-by-one.

Frequently asked questions

Support for GoCD server secrets management interpolation in pipelines as code

Custom source control material plugins like the gocd-git-path-material-plugin only have support for GoCD secrets management and secret variable interpolation (syntax like {{SECRET:[file-secrets][test-password]}}) from GoCD 20.8.0 onwards (changelog).

On earlier versions if you use pipelines-as-code to source control your material definitions, you can however

• use secure_options to source control an encrypted password (see gocd-yaml-config-plugin#pluggable)
• use SSH keys within your GoCD server and agent to authenticate with a private repository
  • If running GoCD on Kubernetes, there is support for doing so based on Kubernetes Secrets within the GoCD Helm chart or any other solution that allows mounting of a file into a container.

Support for triggering from webhooks

Custom source control material plugins like the gocd-git-path-material-plugin have support for triggering from webhooks from GoCD 20.9.0 onwards (changelog). This contrasts with the convention material polling approach (see #27 and gocd/gocd#8170)

Stale data in agent repository clones

This plugin uses Git commands to filter the history and determine the most recent revision that matches the monitored paths. This means that changes that are not monitored in your paths may be "stale". The plugin does not rm un-monitored paths from a clone; meaning your build task could accidentally depend on files in the repository that are out-of-date.

Be careful with your repository structure to keep your monitored path expressions simple, so you can easily reason about whether a given pipeline should have been triggered for a given commit.

Creating new pipelines via UI on pre 19.8.0 GoCD versions

You will not be able to see the Git Path material as a material type when creating a new pipeline. Add a dummy git material, then edit to add a new Git Path material and remove the old one. This problem has been resolved in newer GoCD versions via the pipeline creation wizard.

Contributing

Build

execute the following command to build the plugin

./gradlew clean build

Run

You can quickly test the plugin using Docker Compose, via Docker Desktop, Colima or equivalent.

Execute the following gradle task to start the go-server

./gradlew clean startGoCd

You can now access the go-server via port 8153

reload

If you like to reload the go-server with new build run,

./gradlew clean restartGoCd

stop

You can stop the running docker instance with the following gradle task

./gradlew clean stopGoCd

Release

Released via the gocd-plugin-gradle-task-helpers onto GitHub.

• Check thegocdPlugin.pluginVersion version in build.gradle and bump if necessary
  • The release helpers add a -${COMMIT_NUM} suffix to the version, so this is not strictly necessary for minor changes.
• Tag and publish

  export GITHUB_TOKEN=blah
  PRERELEASE=no ./gradlew clean build githubRelease

• Edit the release description on GitHub if necessary.

Acknowledgements

This plugin contains Git sub-process handling code originally written and released under Apache 2 licenses by

• the GoCD core team
• Srinivas Upadhya via https://github.com/srinivasupadhya/git-cmd/
• Ashwanth Kumar and Chad Wilson via https://github.com/ashwanthkumar/git-cmd