Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flaky kudu tests: schema/table not found and enable CI #11203

Closed
grantatspothero opened this issue Feb 25, 2022 · 6 comments
Closed

Fix flaky kudu tests: schema/table not found and enable CI #11203

grantatspothero opened this issue Feb 25, 2022 · 6 comments
Assignees
@grantatspothero
Labels
bug Something isn't working

Comments

@grantatspothero
Copy link
Contributor

grantatspothero commented Feb 25, 2022 •
edited
Loading

Kudu tests are flaky, see example flake here:
https://github.com/trinodb/trino/runs/5303491394?check_suite_focus=true

Error:  io.trino.plugin.kudu.KuduSmokeTests$TestKuduWithDisabledInferSchemaConnectorTest.init  Time elapsed: 55.908 s  <<< FAILURE!
java.lang.RuntimeException: Table 'default.customer' not found
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:122)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:505)
	at io.trino.testing.QueryAssertions.copyTable(QueryAssertions.java:339)
	at io.trino.testing.QueryAssertions.copyTable(QueryAssertions.java:331)
	at io.trino.testing.QueryAssertions.copyTpchTables(QueryAssertions.java:323)
	at io.trino.plugin.kudu.KuduQueryRunnerFactory.createKuduQueryRunnerTpch(KuduQueryRunnerFactory.java:107)
	at io.trino.plugin.kudu.KuduQueryRunnerFactory.createKuduQueryRunnerTpch(KuduQueryRunnerFactory.java:83)
	at io.trino.plugin.kudu.AbstractKuduConnectorTest.createQueryRunner(AbstractKuduConnectorTest.java:54)
@grantatspothero grantatspothero self-assigned this Feb 25, 2022
@grantatspothero
Copy link
Contributor Author

PR to address kudu flakes here: #11185

@findepi findepi added the bug Something isn't working label Feb 25, 2022
@grantatspothero grantatspothero mentioned this issue Feb 28, 2022
Closed
@hashhar hashhar mentioned this issue Mar 1, 2022
Merged
@ebyhr ebyhr mentioned this issue Mar 1, 2022
Closed
@grantatspothero grantatspothero changed the title Flaky kudu tests Flaky kudu tests: schema/table not found Mar 1, 2022
This was referenced Mar 1, 2022
Merged
Merged
@grantatspothero
Copy link
Contributor Author

And a followup PR is here: #11264

grantatspothero added a commit to grantatspothero/trino that referenced this issue Mar 3, 2022
@grantatspothero
Temporarily disable kudu tests from running in CI
8e605a3 
Flakiness is happening often and impacting other PRs.
See issue here: trinodb#11203
@grantatspothero grantatspothero mentioned this issue Mar 3, 2022
Merged
@ebyhr
Copy link
Member

ebyhr commented Mar 4, 2022 •
edited
Loading

@grantatspothero I believe Kudu test became flaky due to recent changes. Do you know what is the trigger? BaseConnectorTest migration or client upgrade?

hashhar pushed a commit that referenced this issue Mar 4, 2022
@grantatspothero @hashhar
Temporarily disable kudu tests from running in CI
9cf6806 
Flakiness is happening often and impacting other PRs.
See issue here: #11203
@ebyhr
Copy link
Member

ebyhr commented Mar 4, 2022

Labeled RELEASE-BLOCKER since we disabled all Kudu tests in #11313

@ebyhr ebyhr changed the title Flaky kudu tests: schema/table not found Flaky kudu tests: schema/table not found and enable CI Mar 4, 2022
@ebyhr ebyhr changed the title Flaky kudu tests: schema/table not found and enable CI Fix flaky kudu tests: schema/table not found and enable CI Mar 4, 2022
@grantatspothero
Copy link
Contributor Author

grantatspothero commented Mar 4, 2022 •
edited
Loading

@grantatspothero I believe Kudu test became flaky due to recent changes. Do you know what is the trigger? BaseConnectorTest migration or client upgrade?

I unfortunately have never been able to reproduce the flakes on my local machine which makes hunting down the precise regression commit(s) difficult.

But yes:

  • Migrate Kudu to use BaseConnectorTest #11113 Migration to baseconnectortests exposed some existing errors. However, some tests which do not use baseconnectortests TestKuduIntegrationDecimalColumns are flaky so that cannot be the only cause of flakes
  • Upgrade kudu client to 1.15.0 #10940 added better handling of errors during kudu operations. Now we verify operations succeed instead of firing and forgetting. However, some tests are failing not during applying operations but when opening a table like in io.trino.plugin.kudu.TestKuduLatestWithDisabledInferSchemaConnectorTest.init so that cannot be the only cause of flakes

@hashhar
Copy link
Member

hashhar commented Mar 25, 2022

Fixed by #11264

@hashhar hashhar closed this as completed Mar 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@grantatspothero grantatspothero

Labels
bug Something isn't working
Milestone
No milestone
Development

No branches or pull requests
4 participants
@findepi @ebyhr @hashhar @grantatspothero