VariantStorageManagerException exception : status == TILEDB_OK

[chandrans]

Using 4.0.4.0 and got the following rather unhelpful error from a ImportGenomicsDB. I suspect it might be a variant in one of my VCFs being imported causing an issue but the error message doesn't even hint at which file it was working on.

~/gatk-4.0.4.0/gatk GenomicsDBImport -R hs37d5.fa --sample-name-map gvcfs.samplemap --genomicsdb-workspace-path outputsb.workspace -L 6:29691241-33054015 --batch-size 50 --consolidate true
Using GATK jar /home/cloud-user/gatk-4.0.4.0/gatk-package-4.0.4.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/cloud-user/gatk-4.0.4.0/gatk-package-4.0.4.0-local.jar GenomicsDBImport -R hs37d5.fa --sample-name-map gvcfs.samplemap --genomicsdb-workspace-path outputsb.workspace -L 6:29691241-33054015 --batch-size 50 --consolidate true
17:01:47.406 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/cloud-user/gatk-4.0.4.0/gatk-package-4.0.4.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
17:01:47.522 INFO  GenomicsDBImport - ------------------------------------------------------------
17:01:47.522 INFO  GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.0.4.0
17:01:47.522 INFO  GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
17:01:47.522 INFO  GenomicsDBImport - Executing as cloud-user@lustre-assembly-1 on Linux v3.10.0-862.3.3.el7.x86_64 amd64
17:01:47.522 INFO  GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_171-b10
17:01:47.523 INFO  GenomicsDBImport - Start Date/Time: 17 July 2018 17:01:47 BST
17:01:47.523 INFO  GenomicsDBImport - ------------------------------------------------------------
17:01:47.523 INFO  GenomicsDBImport - ------------------------------------------------------------
17:01:47.523 INFO  GenomicsDBImport - HTSJDK Version: 2.14.3
17:01:47.523 INFO  GenomicsDBImport - Picard Version: 2.18.2
17:01:47.523 INFO  GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 2
17:01:47.523 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
17:01:47.523 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
17:01:47.523 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
17:01:47.523 INFO  GenomicsDBImport - Deflater: IntelDeflater
17:01:47.523 INFO  GenomicsDBImport - Inflater: IntelInflater
17:01:47.523 INFO  GenomicsDBImport - GCS max retries/reopens: 20
17:01:47.523 INFO  GenomicsDBImport - Using google-cloud-java patch 6d11bef1c81f885c26b2b56c8616b7a705171e4f from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes
17:01:47.524 INFO  GenomicsDBImport - Initializing engine
17:01:47.959 INFO  IntervalArgumentCollection - Processing 3362775 bp from intervals
17:01:47.984 INFO  GenomicsDBImport - Done initializing engine
Created workspace /mnt/chla/outputsb.workspace
17:01:48.120 INFO  GenomicsDBImport - Vid Map JSON file will be written to outputsb.workspace/vidmap.json
17:01:48.120 INFO  GenomicsDBImport - Callset Map JSON file will be written to outputsb.workspace/callset.json
17:01:48.120 INFO  GenomicsDBImport - Complete VCF Header will be written to outputsb.workspace/vcfheader.vcf
17:01:48.120 INFO  GenomicsDBImport - Importing to array - outputsb.workspace/genomicsdb_array
17:01:48.136 INFO  ProgressMeter - Starting traversal
17:01:48.136 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Batches Processed   Batches/Minute
17:01:48.250 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
17:37:58.214 INFO  ProgressMeter -           6:29691241             36.2                     1              0.0
17:37:58.215 INFO  GenomicsDBImport - Done importing batch 1/143
17:37:58.316 INFO  GenomicsDBImport - Importing batch 2 with 50 samples
18:14:11.907 INFO  ProgressMeter -           6:29691241             72.4                     2              0.0
18:14:11.907 INFO  GenomicsDBImport - Done importing batch 2/143

SNIP

04:51:49.665 INFO  GenomicsDBImport - Importing batch 61 with 50 samples
terminate called after throwing an instance of 'VariantStorageManagerException'
  what():  VariantStorageManagerException exception : status == TILEDB_OK

This Issue was generated from your [forums]
[forums]: https://gatkforums.broadinstitute.org/gatk/discussion/12458/variantstoragemanagerexception-exception-status-tiledb-ok/p1

[chandrans]

Hey Chris. Tagging you for some triage help. User has reported "Figured it out, this is what you get when you run out of disk space. Please could this error message be made more helpful please?" Is there anything you can do? A better error message would help, but not sure who to assign this to.

[cmnbroad]

@kgururaj Can you take a look and see if we can issue a more informative error message.

[kgururaj]

Error reporting was improved in #4645. It should at least print out - "file could not be opened etc"

[lbergelson]

Improved error message sounds like a good solution, thank you.

[lbergelson]

Reopening, this will be closed by a @nalinigans PR in the near future.

[ldgauthier]

There have been a lot of GDB improvements since then. If this is still an issue in the latest GATK we can reopen, but there's no test data here.