CI builds are failing

[terrytangyuan]

test-functional: https://github.com/argoproj/argo-workflows/runs/7564981459?check_suite_focus=true

Condition "to be done" met after 8s
Checking expectation http-template-par-cp6q7
http-template-par-cp6q7 : Failed child 'http-template-par-cp6q7-3675079386' failed
    agent_test.go:65: 
        	Error Trace:	/home/runner/work/argo-workflows/argo-workflows/test/e2e/agent_test.go:65
        	            				/home/runner/work/argo-workflows/argo-workflows/test/e2e/then.go:68
        	            				/home/runner/work/argo-workflows/argo-workflows/test/e2e/then.go:43
        	            				/home/runner/work/argo-workflows/argo-workflows/test/e2e/agent_test.go:64
        	Error:      	Not equal: 
        	            	expected: "Succeeded"
        	            	actual  : "Failed"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(v1alpha1.WorkflowPhase) (len=9) "Succeeded"
        	            	+(v1alpha1.WorkflowPhase) (len=6) "Failed"
        	            	 
        	Test:       	TestAgentSuite/TestParallel
=== FAIL: AgentSuite/TestParallel
FAIL	github.com/argoproj/argo-workflows/v3/test/e2e	8.921s
FAIL
make: *** [Makefile:523: test-functional] Error 1

test-examples: https://github.com/argoproj/argo-workflows/runs/7564908634?check_suite_focus=true

Running examples/http-success-condition.yaml...
workflow.argoproj.io/http-template-condition-vtmg6 created
error: timed out waiting for the condition on workflows/http-template-condition-vtmg6
make: *** [Makefile:527: test-examples] Error 1
Error: Process completed with exit code 2.

[terrytangyuan]

@sarabala1979 @juliev0 Would these be caused by our recent changes?

[alexec]

@sarabala1979 @juliev0 can I please ask for this to be made top priority? It impact all builds for all users. It looks like there hasn't been a build of "CI" on master that passed first time for sometime.

[juliev0]

@terrytangyuan Is this particular failure happening every time? or most times? Which code change are you thinking caused it?

[terrytangyuan]

Yes it happens on master branch as well as on PRs. I tried re-run but they still failed. Not sure which code caused it yet.

[juliev0]

@sarabala1979 says he'll fix it.

[juliev0]

@alexec Yes, getting these false failures addressed is important. That's why I logged these issues: #9027
#9146

Bala and Dillen actually took steps recently to help with some of them. For Unit tests, Bala made it so they run in parallel so they don't time out. For e2e, Bala separated the test-functional suite into two so it's less likely to time out and Dillen increased the time out time.

[sarabala1979]

Failure caused by external URL returns 503. I will fix it before Monday.

[juliev0]

Failure caused by external URL returns 503. I will fix it before Monday.

Thank you, Bala!

[terrytangyuan]

Seems like tests are passing now. What did you do to fix it?

[terrytangyuan]

In case you are looking into this as well, I am seeing issues with docs build but it's caused by StackOverflow. #9258 (comment)