airbyte-ci: failure status to GHA step #42077

clnoll · 2024-07-17T18:58:42Z

When running tests, test failures aren't being surfaced in the GHA.

This PR changes that so we now get a failure status for the workflow if manually-run regression tests fail.

The status check for an individual regression test run can be updated using /approve-regression-tests connector-name=<name>. The global regression test status check can still be updated using /approve-regression-tests with no connector name argument.

Closes #41989.

vercel · 2024-07-17T18:58:46Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jul 24, 2024 3:32pm

clnoll · 2024-07-17T19:33:03Z

@alafanechere this is different than the status check option that we talked about because the live/regression tests are only run manually right now, so there's not necessarily a PR with a status check that can be updated. This feels like the fastest/easiest way to surface the failure.

alafanechere

The reason the workflow does not exit with 1 when a test fail is because I find it useful to differentiate error in workflow / airbyte-ci execution vs test failure.
We (as tooling team) should guarantee that this workflow is stable, if it's failing it means we have something to fix.
We have the global status check for connectors which has the role of showing global failure or success of CI for modified connectors on the PR

clnoll · 2024-07-18T00:30:01Z

Fair enough @alafanechere. Updating to use status checks instead. (Not quite finished testing yet.)

clnoll · 2024-07-21T10:12:22Z

/approve-regression-tests

Check job output.

✅ Approving regression tests

clnoll · 2024-07-21T13:13:57Z

/approve-regression-tests

Check job output.

✅ Approving regression tests

clnoll · 2024-07-21T13:15:22Z

/approve-regression-tests ref=9e48bc9226d17e6e6fd869fcc893b7b143425412

Error: No ref found for: 9e48bc9

clnoll · 2024-07-22T01:29:16Z

/approve-regression-tests ref=catherine/live-tests-status-to-gha

Check job output.

✅ Approving regression tests

clnoll · 2024-07-22T16:56:33Z

/approve-regression-tests ref=catherine/live-tests-status-to-gha

Check job output.

✅ Approving regression tests

clnoll · 2024-07-22T16:59:57Z

/approve-regression-tests connector-name=source-faker ref=catherine/live-tests-status-to-gha

Check job output.

✅ Approving regression tests

clnoll · 2024-07-22T17:11:51Z

/approve-regression-tests connector-name=source-faker ref=catherine/live-tests-status-to-gha

Check job output.

✅ Approving regression tests

clnoll · 2024-07-22T17:16:26Z

/approve-regression-tests connector-name=source-faker ref=catherine/live-tests-status-to-gha

Check job output.

✅ Approving regression tests

clnoll · 2024-07-22T17:31:11Z

airbyte-ci/connectors/pipelines/pipelines/models/contexts/pipeline_context.py

@@ -218,6 +218,13 @@ def dagger_cloud_url(self) -> Optional[str]:
    def remote_storage_enabled(self) -> bool:
        return self.is_ci and bool(self.ci_report_bucket) and bool(self.ci_gcp_credentials)

+    def _should_send_status_check(self) -> bool:
+        should_send = self.is_pr or any(
+            self.pipeline_name.startswith(override) for override in MANUAL_PIPELINE_STATUS_CHECK_OVERRIDE_PREFIXES


@alafanechere I don't love this way of doing things but found that if I didn't override this here, I had to mark the regression test run as a PR. That created duplicated status checks: one called Regression Tests and one called Regression Tests on <source>. Regression Tests on <source> is what's created here, and has links to the report etc. so it's the only one I really want. If you have a suggestion of a nicer way to do this let me know.

I also found that Regression Tests on <source> wasn't getting sent unless the global status check argument was present, which I found surprising. Is that intentional?

That created duplicated status checks
This is intentional for usual connector testing: one global check (required) for all connectors being tested on the PR, and one for each connectors modified on the pr.
IMO you should make the Regression Tests one the required check (to ensure that it has been run on all modified connectors). Regression Tests on <source> could be just used to access reports. The tricky thing would be to mark Regression Tests pass even if it's not run on non certified connectors which could be modified in the same PR as a certified connector.

I also found that Regression Tests on wasn't getting sent unless the global status check argument was present, which I found surprising. Is that intentional?

I don't think it's intentional, did you get the reason why?

alafanechere

I'm not sure I get the necessity of 3 different status check.

Can we have a single required one:

When no connector is modified on a PR: connectors_tests.yml marks it as successful with inline GitHub API calls.
When connectors are modified:
- If certified connectors in modified connectors: airbyte-ci connectors test sets it as pending (at the same level where the global status check is sent)
- If no certified connectors are modified: airbyte-ci connectors testmarks it as successful.
The slash command marks it as successful

Am I missing something or getting your intended flow wrong?

I don't think its required to make regression_tests.yml change the status as results are not deterministics ATM.

alafanechere · 2024-07-24T08:14:46Z

.github/workflows/approve-regression-tests-command.yml

@@ -60,9 +63,32 @@ jobs:
            [1]: ${{ steps.job-vars.outputs.run-url }}

      - name: Approve regression tests
-        id: approve
+        id: approve-regression-tests


I've the impression we have three different context values:

Regression Tests on ${{ github.event.inputs.connector-name }}

Regression Test Results Reviewed and Approved

Regression tests (set in the subcommand of regression_tests.yml

Can you share how these three articulate, I'm not sure I get why we need 3 different ones.

alafanechere · 2024-07-24T08:23:12Z

airbyte-ci/connectors/pipelines/pipelines/models/contexts/pipeline_context.py

@@ -218,6 +218,13 @@ def dagger_cloud_url(self) -> Optional[str]:
    def remote_storage_enabled(self) -> bool:
        return self.is_ci and bool(self.ci_report_bucket) and bool(self.ci_gcp_credentials)

+    def _should_send_status_check(self) -> bool:
+        should_send = self.is_pr or any(
+            self.pipeline_name.startswith(override) for override in MANUAL_PIPELINE_STATUS_CHECK_OVERRIDE_PREFIXES


That created duplicated status checks
This is intentional for usual connector testing: one global check (required) for all connectors being tested on the PR, and one for each connectors modified on the pr.
IMO you should make the Regression Tests one the required check (to ensure that it has been run on all modified connectors). Regression Tests on <source> could be just used to access reports. The tricky thing would be to mark Regression Tests pass even if it's not run on non certified connectors which could be modified in the same PR as a certified connector.

I also found that Regression Tests on wasn't getting sent unless the global status check argument was present, which I found surprising. Is that intentional?

I don't think it's intentional, did you get the reason why?

This reverts commit bc4cd3d0b8f1806b77d27c2feb88e4ef8ae51f78.

octavia-squidington-iii added area/connectors Connector related issues connectors/source/faker labels Jul 17, 2024

clnoll requested a review from alafanechere July 17, 2024 19:31

alafanechere requested changes Jul 17, 2024

View reviewed changes

clnoll marked this pull request as draft July 18, 2024 00:48

octavia-squidington-iii added the connectors/source/salesforce label Jul 18, 2024

clnoll force-pushed the catherine/live-tests-status-to-gha branch 2 times, most recently from 502cf5a to 8ada4e8 Compare July 21, 2024 12:58

clnoll force-pushed the catherine/live-tests-status-to-gha branch 4 times, most recently from dda679f to 6da2e07 Compare July 22, 2024 16:39

clnoll force-pushed the catherine/live-tests-status-to-gha branch from c3573a4 to c042af8 Compare July 22, 2024 17:03

clnoll force-pushed the catherine/live-tests-status-to-gha branch from c042af8 to 9b5a7ef Compare July 22, 2024 17:16

octavia-squidington-iii removed the area/connectors Connector related issues label Jul 22, 2024

clnoll requested a review from alafanechere July 22, 2024 17:26

clnoll marked this pull request as ready for review July 22, 2024 17:26

clnoll commented Jul 22, 2024

View reviewed changes

clnoll force-pushed the catherine/live-tests-status-to-gha branch from 1391175 to 3e38569 Compare July 22, 2024 21:13

natikgadzhi added the team/tooling label Jul 23, 2024

alafanechere requested changes Jul 24, 2024

View reviewed changes

alafanechere approved these changes Jul 24, 2024

View reviewed changes

clnoll added 6 commits July 24, 2024 11:30

airbyte-ci: add status check for regression test runs

6e75983

fake for testing

44964bb

logging

f502b62

fixup! airbyte-ci: add status check for regression test runs

6d08b3c

Revert "fake for testing"

602f584

This reverts commit bc4cd3d0b8f1806b77d27c2feb88e4ef8ae51f78.

version bump

ece04fd

clnoll force-pushed the catherine/live-tests-status-to-gha branch from 3e38569 to ece04fd Compare July 24, 2024 15:32

clnoll merged commit d3a98a6 into master Jul 24, 2024
33 checks passed

clnoll deleted the catherine/live-tests-status-to-gha branch July 24, 2024 16:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airbyte-ci: failure status to GHA step #42077

airbyte-ci: failure status to GHA step #42077

clnoll commented Jul 17, 2024 •

edited

Loading

vercel bot commented Jul 17, 2024 •

edited

Loading

clnoll commented Jul 17, 2024

alafanechere left a comment

clnoll commented Jul 18, 2024

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll Jul 22, 2024

alafanechere Jul 24, 2024

alafanechere left a comment

alafanechere Jul 24, 2024

alafanechere Jul 24, 2024

airbyte-ci: failure status to GHA step #42077

airbyte-ci: failure status to GHA step #42077

Conversation

clnoll commented Jul 17, 2024 • edited Loading

vercel bot commented Jul 17, 2024 • edited Loading

clnoll commented Jul 17, 2024

alafanechere left a comment

Choose a reason for hiding this comment

clnoll commented Jul 18, 2024

clnoll commented Jul 21, 2024 • edited by github-actions bot Loading

clnoll commented Jul 21, 2024 • edited by github-actions bot Loading

clnoll commented Jul 21, 2024 • edited by github-actions bot Loading

clnoll commented Jul 22, 2024 • edited by github-actions bot Loading

clnoll commented Jul 22, 2024 • edited by github-actions bot Loading

clnoll commented Jul 22, 2024 • edited by github-actions bot Loading

clnoll commented Jul 22, 2024 • edited by github-actions bot Loading

clnoll commented Jul 22, 2024 • edited by github-actions bot Loading

clnoll Jul 22, 2024

Choose a reason for hiding this comment

alafanechere Jul 24, 2024

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere Jul 24, 2024

Choose a reason for hiding this comment

alafanechere Jul 24, 2024

Choose a reason for hiding this comment

clnoll commented Jul 17, 2024 •

edited

Loading

vercel bot commented Jul 17, 2024 •

edited

Loading

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 21, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading

clnoll commented Jul 22, 2024 •

edited by github-actions bot

Loading