Adopt go-changelog in Nomad (#10825)

Adopts [`go-changelog`](https://github.com/hashicorp/go-changelog) for managing Nomad's changelog. `go-changelog` is becoming the HashiCorp defacto standard tool for managing changelog, e.g. [Consul](hashicorp/consul#8387), [Vault](hashicorp/vault#10363), [Waypoint](hashicorp/waypoint#1179). [Consul](hashicorp/consul#8387) seems to be the first product to adopt it, and its PR has the most context - though I've updated `.changelog/README.md` with the relevant info here.

## Changes to developers workflow

When opening PRs, developers should add a changelog entry in `.changelog/<PR#>.txt`. Check [`.changelog/README.md`](https://github.com/hashicorp/nomad/blob/docs-adopt-gochangelog/.changelog/README.md#developer-guide). 

For the WIP release, entries can be amended even after the PR merged, and new files may be added post-hoc (e.g. during transition period, missed accidentally, community PRs, etc).

### Transitioning

Pending PRs can start including the changelog entry files immediately.

For 1.1.3/1.0.9 cycle, the release coordinator should create the entries for any PR that gets merged without a changelog entry file. They should also move any 1.1.3 entry in CHANGELOG.md to a changelog entry file, as this PR done for GH-10818.

## Changes to release process

Before cutting a release, release coordinator should update the changelog by inserting the output of `make changelog` to CHANGELOG.md with appropriate headers. See [`.changelog/README.md`](https://github.com/hashicorp/nomad/blob/docs-adopt-gochangelog/.changelog/README.md#how-to-generate-changelog-entries-for-release) for more details.


## Details

go-changelog is a basic templating engine for maintaining changelog in HashiCorp environment.

It expects the changelog entries as files indexed by their PR number. The CLI generates the changelog section for a release by comparing two git references (e.g. `HEAD` and the latest release, e.g. `v1.1.2`), and still requires manual process for updating CHANGELOG.md and final formatting.

The approach has many nice advantages:
* Avoids changelog related merge conflicts: Each PR touches different file!
* Copes with amendments and post-PR updates: Just add or update a changelog entry file using the original PR numbers.
* Addresses the release backporting scenario: Cherry-picking PRs will cherry-pick the relevant changelog entry automatically!
* Only relies on data available through `git` - no reliance on GitHub metadata or require GitHub credentials

The approach has few downsides though:
* CHANGELOG.md going stale during development and must be updated manually before cutting the release
  * Repository watchers can no longer glance at the CHANGELOG.md to see upcoming changes
  * We can periodically update the file, but `go-changelog` tool does not aid with that
* `go-changelog` tool does not offer good error reporting. If an entry is has an invalid tag (e.g. uses `release-note:bugfix` instead of `release-note:bug`), the entry will be dropped silently
  * We should update go-changelog to warn against unexpected entry tags
  * TODO: Meanwhile, PR reviewers and release coordinators should watch out

## Potential follow ups

We should follow up with CI checks to ensure PR changes include a warning. I've opted not to include that now. We still make many non-changelog-worth PRs for website/docs, for large features that get merged in multiple small PRs. I did not want to include a check that fails often.

Also, we should follow up to have `go-changelog` emit better warnings on unexpected tag.

.changelog/10804.txt

@@ -0,0 +1,4 @@
+```release-note:improvement
+consul/connect: automatically set CONSUL_TLS_SERVER_NAME for connect native tasks
+```
+

.changelog/10818.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+csi: fixed a CLI panic when formatting `volume status` with `-verbose` flag
+```

.changelog/10822.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+cli: Fixed system commands, so they correctly use passed flags
+```

.changelog/10823.txt

@@ -1,3 +1,3 @@
 ```release-note:bug
-docker: Moved the generated `/etc/hosts` file's mount source to the allocation directory so that it can be shared between tasks of an allocation.
+driver/docker: Moved the generated `/etc/hosts` file's mount source to the allocation directory so that it can be shared between tasks of an allocation.
 ```

.changelog/changelog.tmpl

@@ -0,0 +1,56 @@
+{{- if .NotesByType.feature }}
+FEATURES:
+
+{{range .NotesByType.feature -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if index .NotesByType "breaking-change" -}}
+BREAKING CHANGES:
+
+{{range index .NotesByType "breaking-change" -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if .NotesByType.security }}
+SECURITY:
+
+{{range .NotesByType.security -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if .NotesByType.improvement }}
+IMPROVEMENTS:
+
+{{range .NotesByType.improvement -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if .NotesByType.deprecation }}
+DEPRECATIONS:
+
+{{range .NotesByType.deprecation -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if .NotesByType.bug }}
+BUG FIXES:
+
+{{range .NotesByType.bug -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+
+{{- if .NotesByType.note }}
+NOTES:
+
+{{range .NotesByType.note -}}
+* {{ template "note" . }}
+{{ end -}}
+{{- end -}}
+

.changelog/note.tmpl

@@ -0,0 +1,3 @@
+{{- define "note" -}}
+{{.Body}}{{if not (stringHasPrefix .Issue "_")}} [[GH-{{- .Issue -}}](https://github.com/hashicorp/nomad/issues/{{- .Issue -}})]{{end}}
+{{- end -}}

CHANGELOG.md

@@ -1,12 +1,5 @@
 ## 1.1.3 (Unreleased)
 
-BUG FIXES:
-* cli: Fixed system commands, so they correctly use passed flags. [[GH-10822](https://github.com/hashicorp/nomad/pull/10822)]
-* csi: Fixed a CLI panic when formatting `volume status` with `-verbose` flag. [[GH-10818](https://github.com/hashicorp/nomad/issues/10818)]
-
-IMPROVEMENTS:
-* consul/connect: automatically set CONSUL_TLS_SERVER_NAME for connect native tasks [[GH-10804](https://github.com/hashicorp/nomad/issues/10804)]
-
 ## 1.1.2 (June 22, 2021)
 
 IMPROVEMENTS:

GNUmakefile

@@ -25,6 +25,10 @@ endif
 # tag corresponding to latest release we maintain backward compatibility with
 PROTO_COMPARE_TAG ?= v1.0.3$(if $(findstring ent,$(GO_TAGS)),+ent,)
 
+# LAST_RELEASE is the git sha of the latest release corresponding to this branch. main should have the latest
+# published release, but backport branches should point to the parent tag (e.g. 1.0.8 in release-1.0.9 after 1.1.0 is cut).
+LAST_RELEASE ?= v1.1.2
+
 default: help
 
 ifeq ($(CI),true)
@@ -99,7 +103,6 @@ bootstrap: deps lint-deps git-hooks # Install all dependencies
 
 .PHONY: deps
 deps:  ## Install build and development dependencies
-## Keep versions in sync with tools/go.mod for now (see https://github.com/golang/go/issues/30515)
 	@echo "==> Updating build dependencies..."
 	go install github.com/hashicorp/go-bindata/go-bindata@bf7910af899725e4938903fb32048c7c0b15f12e
 	go install github.com/elazarl/go-bindata-assetfs/go-bindata-assetfs@234c15e7648ff35458026de92b34c637bae5e6f7
@@ -109,6 +112,7 @@ deps:  ## Install build and development dependencies
 	go install github.com/golang/protobuf/protoc-gen-go@v1.3.4
 	go install github.com/hashicorp/go-msgpack/codec/codecgen@v1.1.5
 	go install github.com/bufbuild/buf/cmd/buf@v0.36.0
+	go install github.com/hashicorp/go-changelog/cmd/changelog-build@latest
 
 .PHONY: lint-deps
 lint-deps: ## Install linter dependencies
@@ -199,10 +203,10 @@ generate-examples: command/job_init.bindata_assetfs.go
 command/job_init.bindata_assetfs.go: command/assets/*
 	go-bindata-assetfs -pkg command -o command/job_init.bindata_assetfs.go ./command/assets/...
 
-.PHONY: changelogfmt
-changelogfmt:
-	@echo "--> Making [GH-xxxx] references clickable..."
-	@sed -E 's|([^\[])\[GH-([0-9]+)\]|\1[[GH-\2](https://github.com/hashicorp/nomad/issues/\2)]|g' CHANGELOG.md > changelog.tmp && mv changelog.tmp CHANGELOG.md
+changelog:
+	@changelog-build -last-release $(LAST_RELEASE) -this-release HEAD \
+		-entries-dir .changelog/ -changelog-template ./.changelog/changelog.tmpl -note-template ./.changelog/note.tmpl
+
 
 ## We skip the terraform directory as there are templated hcl configurations
 ## that do not successfully compile without rendering
@@ -230,7 +234,7 @@ dev: GOOS=$(shell go env GOOS)
 dev: GOARCH=$(shell go env GOARCH)
 dev: GOPATH=$(shell go env GOPATH)
 dev: DEV_TARGET=pkg/$(GOOS)_$(GOARCH)/nomad
-dev: changelogfmt hclfmt ## Build for the current development platform
+dev: hclfmt ## Build for the current development platform
 	@echo "==> Removing old development build..."
 	@rm -f $(PROJECT_ROOT)/$(DEV_TARGET)
 	@rm -f $(PROJECT_ROOT)/bin/nomad

contributing/CHANGELOG.md

@@ -0,0 +1,134 @@
+# How To Use
+
+Nomad uses [`go-changelog`](https://github.com/hashicorp/go-changelog) to generate its changelog on release.
+
+To install, run the following command:
+
+```
+go install github.com/hashicorp/go-changelog/cmd/changelog-build@latest
+```
+
+## Developer Guide
+
+PRs with code changes should include a `.changelog/<PR Number>.txt` file highlighting the user visible change.
+
+The changelog entry should include the affected component (e.g. `driver/docker`, `api`), and a user-centric summary of the change. If an issue warrants more elaborate description, also update the [Upgrade Guide](../website/content/docs/upgrade/upgrade-specific.mdx).
+
+Use the PR number as the file name. Enterprise private changes should have an entry in the OSS repository referencing a public GitHub issues detailing the change and impact.
+
+The `.txt` files have a markdown-like syntax, with a tag signifying the entry type.
+
+Below are some examples of how to generate a CHANGELOG entry with your pull request.
+
+### Improvement
+
+~~~
+```release-note:improvement
+internal/server: Add new option for configs
+```
+~~~
+
+### Feature
+
+Significant major release features. Typically included in release blog announcement and keynotes.
+
+~~~
+```release-note:feature
+**Consul namespace support (Enterprise)**: Run Nomad-defined services in their HashiCorp Consul namespaces more easily using Nomad Enterpris
+```
+~~~
+
+### Improvements
+
+Improvements are incremental and quality of life enhancement that don't rise to the FEATURES level:
+
+```release-note:improvement
+cli: Added `-monitor` flag to `deployment status` command and automatically monitor deployments from `job run` command.
+```
+
+### Bug
+
+~~~
+```release-note:bug
+client/fingerprint/java: Fixed a bug where java fingerprinter would not detect some Java distributions
+```
+~~~
+
+### Multiple Entries
+
+If a PR has multiple entries,
+
+~~~
+```release-note:bug
+driver/docker: Fix broken code
+```
+```release-note:bug
+client: Fix broken code
+```
+```release-note:bug
+cli: Fix broken code
+```
+~~~
+
+### Long Description with Markdown
+
+~~~
+```release-note:feature
+main: Lorem ipsum dolor `sit amet`, _consectetur_ adipiscing elit, **sed** do
+eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
+veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+```
+~~~
+
+
+## How to generate CHANGELOG entries for release
+
+Below is an example for running `go-changelog` to generate a collection of
+entries. It will generate output that can be inserted into CHANGELOG.md manually.
+
+To generate latest release changelog entry:
+
+```
+make changelog
+```
+
+You can also call `changelog-build` binary directly. For more information as to what each flag does, make sure to run `changelog-build -help`.
+
+```
+cd .changelog
+changelog-build -last-release v1.1.2 -this-release HEAD \
+                -entries-dir . -changelog-template changelog.tmpl -note-template note.tmpl
+```
+
+The command will output the changelog entries to be inserted into CHANGELOG.md with version header (e.g. `## 1.1.3 (August 10, 2022)`). The output is like:
+
+```md
+IMPROVEMENTS:
+
+* Added the `bar` interface. [[GH-2032](https://github.com/hashicorp/nomad/issues/2032)]
+
+DEPRECATIONS:
+
+* Deprecated the `foo` interface, please use the `bar` interface instead. [[GH-1001](https://github.com/hashicorp/nomad/issues/1001)]
+
+BUG FIXES:
+
+* csi: fixed a CLI panic when formatting `volume status` with `-verbose` flag [[GH-10818](https://github.com/hashicorp/nomad/issues/10818)]
+```
+
+## FAQ
+
+### Can I amend the changelog entry after the PR merged?
+
+Yes. If the release is not out, just update the entry file, or add missed entries. If the original PR is a to-be-backported change, ensure the changelog entry update PR also has the `label:stage/needs-backporting` and appropriate milestone so the backport includes the update too.
+
+When amending entries for an already published release, update the CHANGELOG.md file directly.
+
+### Just modified the entry file, but the `go-changelog` isn't picking it up.
+
+Commit the entry first! `go-changelog` operators on committed files only.
+
+