Merge pull request #93 from codefresh-io/CR-8565-sync-v3.2.6

Cr 8565 sync v3.2.6

.github/pull_request_template.md

@@ -1,9 +1,17 @@
-Tips:
+Don't bother creating a PR until you've done this:
+
+* [ ] Run `make pre-commit -B` to fix codegen, lint, and commit message problems.
+
+Create your PR as a draft.
+
 
-* Maybe add you organization to [USERS.md](https://github.com/argoproj/argo-workflows/blob/master/USERS.md).
-* Your PR needs to pass the required checks before it can be approved. If the check is not required (e.g. E2E tests) it does not need to pass
-* Sign-off your commits to pass the DCO check: `git commit --signoff`.
-* Run `make pre-commit -B` to fix codegen or lint problems. 
+* Your PR needs to pass the required checks before it can be approved. If the check is not required (e.g. E2E tests) it
+  does not need to pass.
+* Once required tests have passed, you can make it "Ready for review".
 * Say how how you tested your changes. If you changed the UI, attach screenshots.
-* If changes were requested, and you've made them, then dismis the review to get it looked at again.
-* You can ask for help! 
+
+Tips:
+
+* If changes were requested, and you've made them, then dismiss the review to get it looked at again.
+* Add you organization to [USERS.md](https://github.com/argoproj/argo-workflows/blob/master/USERS.md) if you like.
+* You can ask for help!

.github/workflows/ci-build.yaml

@@ -13,7 +13,7 @@ on:
 jobs:
   tests:
     name: Unit Tests
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     # 5m30
     timeout-minutes: 8
     steps:
@@ -33,16 +33,13 @@ jobs:
       - run: make server/static/files.go STATIC_FILES=false
       - run: go build -v ./...
       - run: make test STATIC_FILES=false GOTEST='go test -covermode=atomic -coverprofile=coverage.out'
+      # engineers just ignore this in PRs, so lets not even run it
       - run: bash <(curl -s https://codecov.io/bash)
+        if: github.ref == 'refs/heads/master'
 
   e2e-tests:
     name: E2E Tests
-    runs-on: ubuntu-20.04
-    # test-api: 7m (1m10s locally)
-    # test-cli: 12m (4m locally)
-    # test-cron: 8m
-    # test-executor 8m (2m locally)
-    # test-functional: 13m (7m locally)
+    runs-on: ubuntu-latest
     timeout-minutes: 20
     env:
       KUBECONFIG: /home/runner/.kubeconfig
@@ -57,14 +54,20 @@ jobs:
             containerRuntimeExecutor: docker
             profile: mysql
           - test: test-cron
-            containerRuntimeExecutor: docker
+            containerRuntimeExecutor: emissary
             profile: minimal
-          - test: test-examples
+          - test: test-executor
+            containerRuntimeExecutor: emissary
+            profile: minimal
+          - test: test-functional
             containerRuntimeExecutor: emissary
             profile: minimal
           - test: test-executor
             containerRuntimeExecutor: docker
             profile: minimal
+          - test: test-examples
+            containerRuntimeExecutor: emissary
+            profile: minimal
           - test: test-executor
             containerRuntimeExecutor: k8sapi
             profile: minimal
@@ -74,9 +77,6 @@ jobs:
           - test: test-executor
             containerRuntimeExecutor: pns
             profile: minimal
-          - test: test-functional
-            containerRuntimeExecutor: docker
-            profile: minimal
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-go@v2
@@ -94,9 +94,27 @@ jobs:
         with:
           path: /home/runner/go/bin
           key: go-bin-v1-${{ hashFiles('**/go.mod') }}
+      - name: Cache Docker layers
+        uses: actions/cache@v2
+        with:
+          path: /tmp/.buildx-cache
+          key: ${{ runner.os }}-buildx-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-buildx-
+      - uses: docker/setup-buildx-action@v1
       - run: mkdir -p /tmp/log/argo-e2e
+      # we never want to pull images by accident on CI after we built them
+      # so imagePullPolicy=Never
+      # but, we can pull it if we don't need it
+      - run: docker pull quay.io/argoproj/argoexec:latest
+        if: ${{!(matrix.test == 'test-executor' || matrix.test == 'test-functional')}}
+      - name: make argoexec-image
+        if: ${{matrix.test == 'test-executor' || matrix.test == 'test-functional'}}
+        # retry this once, as it can be flakey
+        run: |
+          make argoexec-image STATIC_FILES=false || make argoexec-image STATIC_FILES=false
+          docker image prune -f
       - name: Install and start K3S
-        timeout-minutes: 3
         run: |
           curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.21.2+k3s1 INSTALL_K3S_CHANNEL=stable INSTALL_K3S_EXEC=--docker K3S_KUBECONFIG_MODE=644 sh -
           until kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml cluster-info ; do sleep 10s ; done
@@ -113,21 +131,20 @@ jobs:
           echo '127.0.0.1 mysql'    | sudo tee -a /etc/hosts
       - run: make install controller cli $(go env GOPATH)/bin/goreman PROFILE=${{matrix.profile}} E2E_EXECUTOR=${{matrix.containerRuntimeExecutor}} AUTH_MODE=client STATIC_FILES=false LOG_LEVEL=info
       - run: make start PROFILE=${{matrix.profile}} E2E_EXECUTOR=${{matrix.containerRuntimeExecutor}} AUTH_MODE=client STATIC_FILES=false LOG_LEVEL=info UI=false > /tmp/log/argo-e2e/argo.log 2>&1 &
-        timeout-minutes: 4
-      - name: make argoexec-image
-        run: make argoexec-image STATIC_FILES=false
       - run: make wait
         timeout-minutes: 4
       - run: make ${{matrix.test}} E2E_TIMEOUT=1m STATIC_FILES=false
       - name: cat argo.log
         if: ${{ failure() }}
         run: cat /tmp/log/argo-e2e/argo.log
-
+      - name: MinIO logs
+        if: ${{ failure() }}
+        run: kubectl -n argo logs deploy/minio
   codegen:
     name: Codegen
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     needs: [ tests ]
-    timeout-minutes: 9
+    timeout-minutes: 15
     env:
       GOPATH: /home/runner/go
       PROTOC_ZIP: protoc-3.11.1-linux-x86_64.zip
@@ -173,7 +190,7 @@ jobs:
 
   lint:
     name: Lint
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     needs: [ tests, codegen ]
     timeout-minutes: 6
     env:
@@ -187,7 +204,7 @@ jobs:
 
   ui:
     name: UI
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     timeout-minutes: 6
     env:
       NODE_OPTIONS: --max-old-space-size=4096

.github/workflows/gh-pages.yaml

@@ -8,7 +8,7 @@ on:
 jobs:
   deploy:
     if: github.repository == 'argoproj/argo-workflows'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: Setup Python

.github/workflows/release.yaml

@@ -17,7 +17,7 @@ jobs:
   build-linux-amd64:
     name: Build & push linux/amd64
     if: github.repository == 'codefresh-io/argo-workflows'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         platform: [ linux/amd64 ]
@@ -85,7 +85,7 @@ jobs:
   build-linux-arm64:
     name: Build & push linux/arm64
     if: github.repository == 'codefresh-io/argo-workflows'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       matrix:
         platform: [ linux/arm64 ]
@@ -200,7 +200,7 @@ jobs:
   push-images:
     name: Push manifest with all images
     if: github.repository == 'codefresh-io/argo-workflows'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     needs: [ build-linux-amd64, build-linux-arm64, build-windows ]
     steps:
       - uses: actions/checkout@v2
@@ -253,7 +253,7 @@ jobs:
   test-images-linux-amd64:
     name: Try pulling linux/amd64
     if: github.repository == 'codefresh-io/argo-workflows'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     needs: [ push-images ]
     strategy:
       matrix:
@@ -328,7 +328,7 @@ jobs:
           done
 
   publish-release:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     if: github.repository == 'codefresh-io/argo-workflows'
     needs: [ push-images, test-images-linux-amd64, test-images-windows ]
     env:

.github/workflows/sdks.yaml

@@ -3,10 +3,8 @@ on:
   push:
     tags:
       - v*
-      - 'v3.2.*'
-      - 'v3.1.*'
     branches:
-      - dev-*
+      - master
 jobs:
   sdk:
     if: github.repository == 'argoproj/argo-workflows'
@@ -21,9 +19,3 @@ jobs:
       - run: make --directory sdks/${{matrix.name}} publish -B
         env:
           JAVA_SDK_MAVEN_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
-      - uses: peter-evans/create-pull-request@v3
-        with:
-          title: 'chore: updated ${{matrix.name}} SDK'
-          commit-message: 'chore: updated ${{matrix.name}} SDK'
-          branch: create-pull-request/sdk/${{matrix.name}}
-          signoff: true

.golangci.yml

@@ -1,7 +1,7 @@
 # https://golangci-lint.run/usage/quick-start/
 run:
   concurrency: 4
-  timeout: 5m
+  timeout: 8m
   skip-dirs:
     - pkg/client
     - vendor

Makefile

@@ -219,7 +219,13 @@ argoexec-image:
 
 %-image:
 	[ ! -e dist/$* ] || mv dist/$* .
-	docker buildx build -t $(IMAGE_NAMESPACE)/$*:$(VERSION) --target $* --output=type=docker .
+	docker buildx install
+	docker build \
+		-t $(IMAGE_NAMESPACE)/$*:$(VERSION) \
+		--target $* \
+		--cache-from "type=local,src=/tmp/.buildx-cache" \
+		--cache-to "type=local,dest=/tmp/.buildx-cache" \
+		--output=type=docker .
 	[ ! -e $* ] || mv $* dist/
 	docker run --rm -t $(IMAGE_NAMESPACE)/$*:$(VERSION) version
 	if [ $(K3D) = true ]; then k3d image import $(IMAGE_NAMESPACE)/$*:$(VERSION); fi
@@ -396,7 +402,7 @@ test: server/static/files.go dist/argosay
 	env KUBECONFIG=/dev/null $(GOTEST) ./...
 
 .PHONY: install
-install:
+install: githooks
 	kubectl get ns $(KUBE_NAMESPACE) || kubectl create ns $(KUBE_NAMESPACE)
 	kubectl config set-context --current --namespace=$(KUBE_NAMESPACE)
 	@echo "installing PROFILE=$(PROFILE), E2E_EXECUTOR=$(E2E_EXECUTOR)"
@@ -570,15 +576,15 @@ validate-examples: api/jsonschema/schema.json
 	cd examples && go test
 
 # pre-push
+.git/hooks/commit-msg: hack/git/hooks/commit-msg
+	cp -v hack/git/hooks/commit-msg .git/hooks/commit-msg
 
-.PHONY: pre-commit
-pre-commit: codegen lint test start
 
-ifeq ($(GIT_BRANCH),master)
-LOG_OPTS := '-n10'
-else
-LOG_OPTS := 'origin/master..'
-endif
+.PHONY: githooks
+githooks: .git/hooks/commit-msg
+
+.PHONY: pre-commit
+pre-commit: githooks codegen lint
 
 release-notes: /dev/null
 	version=$(VERSION) envsubst < hack/release-notes.md > release-notes

USERS.md

@@ -147,6 +147,7 @@ Currently, the following organizations are **officially** using Argo Workflows:
 1. [WooliesX](https://wooliesx.com.au/)
 1. [Woolworths Group](https://www.woolworthsgroup.com.au/)
 1. [Workiva](https://www.workiva.com/)
+1. [Voyager](https://investvoyager.com/)
 1. [Zhihu](https://www.zhihu.com/)
 
 ### Projects Using Argo

cmd/argo/commands/cron/delete.go

@@ -38,6 +38,6 @@ func NewDeleteCommand() *cobra.Command {
 		},
 	}
 
-	command.Flags().BoolVar(&all, "all", false, "Delete all workflow templates")
+	command.Flags().BoolVar(&all, "all", false, "Delete all cron workflows")
 	return command
 }

cmd/argo/commands/cron/util.go

@@ -11,11 +11,7 @@ import (
 // GetNextRuntime returns the next time the workflow should run in local time. It assumes the workflow-controller is in
 // UTC, but nevertheless returns the time in the local timezone.
 func GetNextRuntime(cwf *v1alpha1.CronWorkflow) (time.Time, error) {
-	cronScheduleString := cwf.Spec.Schedule
-	if cwf.Spec.Timezone != "" {
-		cronScheduleString = "CRON_TZ=" + cwf.Spec.Timezone + " " + cronScheduleString
-	}
-	cronSchedule, err := cron.ParseStandard(cronScheduleString)
+	cronSchedule, err := cron.ParseStandard(cwf.Spec.GetScheduleString())
 	if err != nil {
 		return time.Time{}, err
 	}

docs/CONTRIBUTING.md

@@ -8,13 +8,19 @@ Please [raise an issue in Github](https://github.com/argoproj/argo-workflows/iss
 
 See [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
 
+## Contributor Meetings
+
+A weekly opportunity for committers and maintainers of Workflows, Events, and Dataflow to discuss their current work and talk
+about what’s next. Feel free to join us! For Contributor Meeting information, minutes and recordings
+please [see here](https://bit.ly/argo-data-weekly).
+
 ## How To Contribute
 
 We're always looking for contributors.
 
 * Documentation - something missing or unclear? Please submit a pull request!
 * Code contribution - investigate a [help wanted issue](https://github.com/argoproj/argo-workflows/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22+label%3A%22good+first+issue%22), or anything labelled with "good first issue"?
-* Join the #argo-devs channel on [our Slack](https://argoproj.github.io/community/join-slack).
+* Join the `#argo-contributors` channel on [our Slack](https://argoproj.github.io/community/join-slack).
 
 ### Running Locally
 

docs/cli/argo_cron_delete.md

@@ -9,7 +9,7 @@ argo cron delete [CRON_WORKFLOW... | --all] [flags]
 ### Options
 
 ```
-      --all    Delete all workflow templates
+      --all    Delete all cron workflows
   -h, --help   help for delete
 ```
 

docs/fields.md

@@ -2185,6 +2185,8 @@ Outputs hold parameters, artifacts, and results from a step
 
 - [`dag-conditional-parameters.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/dag-conditional-parameters.yaml)
 
+- [`data-transformations.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/data-transformations.yaml)
+
 - [`exit-handler-with-artifacts.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-with-artifacts.yaml)
 
 - [`exit-handler-with-param.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-with-param.yaml)

docs/node-field-selector.md

@@ -61,7 +61,7 @@ Consider the following workflow:
 
 Here we have two steps with the same `displayName`: `wait-approval`. To select one to suspend, we need to use their
 `name`, either `appr-promotion-ffsv4.app1.wait-approval` or `appr-promotion-ffsv4.app3.wait-approval`. If it is not clear
-what the full name of a done is, it can be found using `kubectl`:
+what the full name of a node is, it can be found using `kubectl`:
 
 ```
 $ kubectl get wf appr-promotion-ffsv4 -o yaml

docs/releases.md

@@ -17,12 +17,13 @@ Both the `argo-server` and `argocli` should be the same version as the controlle
 
 # Release Cycle
 
+For **stable**, use the latest patch version.
 For **unstable**, we build and tag `latest` images for every commit to master.
 
-New major versions are released roughly every 3 months. Release candidates for each major release are typically available
-for 6 weeks before the release becomes generally available.
+New minor versions are released roughly every 3 months. Release candidates for each major release are typically available
+for 4-6 weeks before the release becomes generally available.
 
-Otherwise, we typically release once a week:
+Otherwise, we typically release weekly:
 
-* Patch fixes for the current stable version. These are tagged `stable`.
+* Patch fixes for the current stable version. 
 * The next release candidate, if we are currently in a release-cycle.

docs/security.md

@@ -75,7 +75,7 @@ Argo Workflows requires various levels of network access depending on configurat
 
 The argo server is commonly exposed to end-users to provide users with a user interface for visualizing and managing their workflows. It must also be exposed if leveraging [webhooks](webhooks.md) to trigger workflows. Both of these use cases require that the argo-server Service to be exposed for ingress traffic (e.g. with an Ingress object or load balancer). Note that the Argo UI is also available to be accessed by running the server locally (i.e. `argo server`) using local kubeconfig credentials, and visiting the UI over https://localhost:2746.
 
-The argo server additionally has a feature to allow downloading of artifacts through the user interface. This feature requires that the argo-server be given egress access to the underlying artifact provider (e.g. S3, GCS, MinIO, Arfactory) in order to download and stream the artifact.
+The argo server additionally has a feature to allow downloading of artifacts through the user interface. This feature requires that the argo-server be given egress access to the underlying artifact provider (e.g. S3, GCS, MinIO, Artifactory) in order to download and stream the artifact.
 
 ### Workflow Controller