Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integration tests regularly fail for HDFS #1428

Closed
roeap opened this issue Jun 3, 2023 · 1 comment
Closed

integration tests regularly fail for HDFS #1428

roeap opened this issue Jun 3, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@roeap
Copy link
Collaborator

roeap commented Jun 3, 2023

Description

The integration tests have become quite flaky since we added HDFS support. Since we are right now not adding this as a python feature, and it relies on external object store crate, should we maybe improve the ergonomics of using external stores along with better documentation, and remove that as a direct dependency?

Or does someone have an idea on how to fix the flakiness?

Use Case

Related Issue(s)
@roeap roeap added the enhancement New feature or request label Jun 3, 2023
@roeap roeap mentioned this issue Jun 3, 2023
Merged
@rtyler
Copy link
Member

rtyler commented Jun 3, 2023

Since we are right now not adding this as a python feature, and it relies on external object store crate, should we maybe improve the ergonomics of using external stores along with better documentation, and remove that as a direct dependency

This is certainly my preference for a number of reasons, test flakiness not the least of them. I think our interfaces are not quite there to allow a Python user to Bring Their Own Object Store, but I'm doing a lot more thinking in that direction. I'm wondering if there would be the possibility to split the entire vertical slice (Rust + Python) of HDFS into a supplementary crate in delta-incubator to be worked on and experiment with that interface layer. 🤔

@roeap roeap mentioned this issue Jun 4, 2023
Merged
roeap added a commit that referenced this issue Jun 5, 2023
@roeap
ci: prune CI/CD pipelines (#1433)
427e3d6 
# Description

With our current integration tests being close to unusable due to the
very frequent failures of the HDFS tests, I took the liberty to disable
HDFS in the integration tests for now. While at it I also sprinkled a
couple of `RUSTFLAG`s in the hopes of improving the overall experience
with our CI/CD.

Last but not least, I added python 3.11 to the test matrix, in the hopes
that it may just work :).

# Related Issue(s)
 related #1428
@rtyler rtyler closed this as completed Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rtyler @roeap