Downstream tools [GATK] affected by disq

[SHuang-Broad]

Hello Tom,

I am reporting a behavior here that I observed while running GATK 4 PrintReadsSpark in a vanilla manner.
I do not know how to summarize the behavior so I am simply copying the error message here

A USER ERROR has occurred: Couldn't write file hdfs://shuang-disq-test-m:8020/data/m64020_190208_213731.subreads.ccs.aligned.merged.chr21.bam because writing failed with exception concat: source file /data/m64020_190208_213731.subreads.ccs.aligned.merged.chr21.bam.parts/part-r-00000 is invalid or empty or underConstruction
	at org.apache.hadoop.hdfs.server.namenode.FSDirConcatOp.verifySrcFiles(FSDirConcatOp.java:160)
	at org.apache.hadoop.hdfs.server.namenode.FSDirConcatOp.concat(FSDirConcatOp.java:68)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concat(FSNamesystem.java:1935)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.concat(NameNodeRpcServer.java:1001)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.concat(ClientNamenodeProtocolServerSideTranslatorPB.java:583)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606)

NOTE: the bam is a long read bam (PacBio sequence) produced by minimap2.

If you need more, please let me know.

Thanks!

---

In the attached zip, there's the setup.dataproc.sh for creating the dataproc cluster, printReadsSpark.sh for extracting reads mapped to chr21, and the full stacktrace PrintReadsSpark_Error.log.

PrintReadsSpark_error.zip

[lbergelson]

@tomwhite I was meaning to ask you about this when we spoke just now, but I forgot... Any thoughts about what's going on here?

[tomwhite]

@SHuang-Broad thanks for the error report. The error log says that the concat operation is failing. Do you know if you get the same error if you remove the interval filter (-L chr21)? I wonder if it's creating empty part files that due to the filtering.

[SHuang-Broad]

@tomwhite

I've done several experiments here

• with the same "only chr21" request, but disabling the tool's default filters (leading to the same error as before)
• request reads mapped to chrX (and later chr1 in a separate run), also disabling the tool's default filters (ditto)
• copy the whole BAM with the tool's default read filters in place (passing, good)
• requesting "chr21" reads, on the just-copied bam, which unlike the original bam, now has an accompanying "sbi" index file (again, fail with the same error)

So it seems like requesting -L is causing the trouble?

---

experiments_on_July_8th.zip

[tomwhite]

Thanks @SHuang-Broad! I will try to reproduce and write a fix.

[heuermh]

@SHuang-Broad Thank you for reporting this issue! If the fix merged in #111 doesn't work for you, please reopen this issue or create a new one.

[SHuang-Broad]

@tomwhite @heuermh
Louis and I just tested the patch and it is working as expected.
Thanks!