Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python test_async_io fails on GDS-enabled system #558

Closed
kingcrimsontianyu opened this issue Nov 20, 2024 · 0 comments · Fixed by #559
Closed

Python test_async_io fails on GDS-enabled system #558

kingcrimsontianyu opened this issue Nov 20, 2024 · 0 comments · Fixed by #559
Assignees
Labels
bug Something isn't working

Comments

@kingcrimsontianyu
Copy link
Contributor

kingcrimsontianyu commented Nov 20, 2024

# <my_gds_supported_path> below represents a GDS supported file system.
# Note that in the dev container, the temp directory must not be the default /tmp.
# The default /tmp is an OverlayFS, which would cause cuFile I/O to have "internal errors" at runtime
export TMPDIR=<my_gds_supported_path>
pytest ~/kvikio/python/kvikio/tests/test_async_io.py -x

On a GDS-enabled system, the above test yields the following error:

python/kvikio/tests/test_async_io.py:36: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: cuFile error at: /home/coder/kvikio/cpp/include/kvikio/stream.hpp:136: unsupported file open flags

kvikio/_lib/future.pyx:18: RuntimeError

During handling of the above exception, another exception occurred:

tmp_path = PosixPath('/mnt/profile/pytest-of-coder/pytest-37/test_read_write_1_0'), size = 1

    @pytest.mark.parametrize("size", [1, 10, 100, 1000, 1024, 4096, 4096 * 10])
    def test_read_write(tmp_path, size):
        """Test basic read/write"""
        filename = tmp_path / "test-file"
    
        stream = cupy.cuda.Stream()
    
        # Write file
        a = cupy.arange(size)
        f = kvikio.CuFile(filename, "w")
        assert not f.closed
        assert check_bit_flags(f.open_flags(), os.O_WRONLY)
        assert f.raw_write_async(a, stream.ptr).check_bytes_done() == a.nbytes
    
        # Try to read file opened in write-only mode
>       with pytest.raises(RuntimeError, match="Operation not permitted"):
E       AssertionError: Regex pattern did not match.
E        Regex: 'Operation not permitted'
E        Input: 'cuFile error at: /home/coder/kvikio/cpp/include/kvikio/stream.hpp:136: unsupported file open flags'

python/kvikio/tests/test_async_io.py:33: AssertionError
======================================================== short test summary info =========================================================
FAILED python/kvikio/tests/test_async_io.py::test_read_write[1] - AssertionError: Regex pattern did not match.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================== 1 failed in 2.82s ============================================================
cuFile error at: /home/coder/kvikio/cpp/include/kvikio/stream.hpp:136: unsupported file open flags
@kingcrimsontianyu kingcrimsontianyu added the bug Something isn't working label Nov 20, 2024
@kingcrimsontianyu kingcrimsontianyu self-assigned this Nov 20, 2024
@rapids-bot rapids-bot bot closed this as completed in #559 Nov 21, 2024
@rapids-bot rapids-bot bot closed this as completed in b0e7bd1 Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant