test failures in TestTruncatedBAM

[nieder]

On 0.19.1 (python 3.7-3.10 tested), tests fail like this:

tests/AlignmentFile_test.py::TestHeaderWithProgramOptions::testHeader PASSED
tests/AlignmentFile_test.py::TestTruncatedBAM::testTruncatedBam2 [E::idx_find_and_load] Could not retrieve index file for '/sw/build.build/pysam-py37-0.19.1-1/pysam-0.19.1/tests/pysam_data/ex2_truncated.bam'
FAILED
tests/AlignmentFile_test.py::TestTruncatedBAM::testTruncatedBamIterator [E::idx_find_and_load] Could not retrieve index file for '/sw/build.build/pysam-py37-0.19.1-1/pysam-0.19.1/tests/pysam_data/ex2_truncated.bam'
FAILED
tests/AlignmentFile_test.py::TestCorruptBAM::testCorruptBamIterator [E::idx_find_and_load] Could not retrieve index file for '/sw/build.build/pysam-py37-0.19.1-1/pysam-0.19.1/tests/pysam_data/ex2_corrupt.bam'
PASSED
tests/AlignmentFile_test.py::TestBTagSam::testReadTags PASSED
...
======================================================================== FAILURES ========================================================================
___________________________________________________________ TestTruncatedBAM.testTruncatedBam2 ___________________________________________________________

self = <AlignmentFile_test.TestTruncatedBAM testMethod=testTruncatedBam2>

    def testTruncatedBam2(self):
        self.assertRaises(IOError,
                          pysam.AlignmentFile,
>                         os.path.join(BAM_DATADIR, 'ex2_truncated.bam'))
E       AssertionError: OSError not raised by AlignmentFile

tests/AlignmentFile_test.py:1457: AssertionError
_______________________________________________________ TestTruncatedBAM.testTruncatedBamIterator ________________________________________________________

self = <AlignmentFile_test.TestTruncatedBAM testMethod=testTruncatedBamIterator>

    def testTruncatedBamIterator(self):
        s = pysam.AlignmentFile(os.path.join(BAM_DATADIR, 'ex2_truncated.bam'),
                                ignore_truncation=True)
    
        def iterall(x):
            return len([a for a in x])
>       self.assertRaises(IOError, iterall, s)
E       AssertionError: OSError not raised by iterall

tests/AlignmentFile_test.py:1465: AssertionError

Running samtools index ex2_truncated.bam gets rid of the "can't retrieve index" issue, but the assertion still happens.

[jmarshall]

The ex2_truncated.bam test file is constructed (in tests/pysam_data/Makefile) using this somewhat naive rule:

ex2_truncated.bam: ex2.bam
        head -c 124000 ex2.bam > ex2_truncated.bam

And ex2.bam is constructed via samtools view -bo ex2.bam ex2.sam from the original SAM test file.

Samtools built without libdeflate produces an ex2.bam file of 124703 bytes, so that head command produces an invalidly truncated BAM file.

3½ years after that test case was added, HTSlib added support for using libdeflate in place of zlib. Samtools built with libdeflate produces an ex2.bam file of 122804 bytes. So that head command leaves the file unchanged, and the tests cases fail because ex2_truncated.bam is unexpectedly valid.

What does ls -l tests/pysam_data/ex2*.bam report in your build directory? It would appear that you are the first person to run pysam's tests using a libdeflate-enabled samtools 😄

So we'll make the rule that truncates ex2_truncated.bam rather smarter. In the meantime, if it is important to produce a clear test run, you could work around this by using a non-libdeflate samtools to generate pysam's test data files.

[nieder]

$ ls -l tests/pysam_data/ex2*.bam
-rw-r--r--  1 fink-bld  fink-bld  123300 Aug 27 18:03 tests/pysam_data/ex2.bam
-rw-r--r--  1 fink-bld  fink-bld  123285 Aug 27 18:03 tests/pysam_data/ex2_corrupt.bam
-rw-r--r--  1 fink-bld  fink-bld  123300 Aug 27 18:03 tests/pysam_data/ex2_truncated.bam

Changing 124000 to 110000 makes the 2 tests pass.