Misc cleanup needed for bigdatagenomics/cannoli#65 #1704

fnothaft · 2017-08-30T21:03:55Z

heuermh · 2017-08-30T21:16:02Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/fragment/InterleavedFASTQInFormatter.scala

@@ -59,11 +60,11 @@ class InterleavedFASTQInFormatter private (
    iter.flatMap(frag => {
      val reads = converter.convertFragment(frag).toSeq

-      if (reads.size < 2) {
+      if (enableLogging && reads.size < 2) {


would simply dropping to log.info accomplish the same goal?

I mean, it really is a warning message. However, if you're doing development of a piped tool and you want to read your executor logs to figure out why the tool running under the pipe is blowing up, the warnings get in the way of reading the logs.

If you're not 100% sold on this change, I'm OK with dropping it. It was just for my convenience when developing.

I don't have data to test these particular checks (though I suppose I could invent some) and am not all that confident our logging configuration works. Could you try adding

log4j.logger.org.bdgenomics.adam.rdd.fragment.InterleavedFASTQInFormatter=ERROR

to your log4j.properties and see if it does the trick?

Yeah, agreed. I've rebased this and dropped the commit that resolves #1701.

coveralls · 2017-08-30T21:19:40Z

Coverage decreased (-0.003%) to 83.59% when pulling 69dcfa5 on fnothaft:issues/1701-1702-ifq-cleanup into 511f925 on bigdatagenomics:master.

AmplabJenkins · 2017-08-30T21:27:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2346/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1704/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains ae21c0b # timeout=10Checking out Revision ae21c0b (origin/pr/1704/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f ae21c0b582a262fb3b7087830e45423c876b97cdFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2017-09-06T07:21:03Z

This is good to go from my side.

AmplabJenkins · 2017-09-06T07:41:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2355/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1704/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 2bfd867aff6fd7412a9d05324462bf10b5b8ffa8 # timeout=10Checking out Revision 2bfd867aff6fd7412a9d05324462bf10b5b8ffa8 (origin/pr/1704/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 2bfd867aff6fd7412a9d05324462bf10b5b8ffa8First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

coveralls · 2017-09-06T07:59:48Z

Coverage decreased (-0.009%) to 83.463% when pulling c8550bc on fnothaft:issues/1701-1702-ifq-cleanup into 51efbaf on bigdatagenomics:master.

coveralls · 2017-09-06T07:59:49Z

Coverage decreased (-0.4%) to 83.072% when pulling c8550bc on fnothaft:issues/1701-1702-ifq-cleanup into 51efbaf on bigdatagenomics:master.

AmplabJenkins · 2017-09-06T08:09:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2356/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1704/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 311615c # timeout=10Checking out Revision 311615c (origin/pr/1704/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 311615c4341eb3d25d23ae77095be8381a06005aFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.11,2.1.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,2.1.0,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.1.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.1.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.1.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2017-09-06T14:02:31Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/read/AnySAMOutFormatter.scala

@@ -29,7 +29,7 @@ import scala.collection.mutable.ListBuffer
 * An OutFormatter that automatically infers whether the piped input is SAM or
 * BAM. Autodetecting streamed CRAM is not currently supported.
 */
-class AnySAMOutFormatter extends OutFormatter[AlignmentRecord] {
+case class AnySAMOutFormatter(stringency: ValidationStringency = ValidationStringency.STRICT) extends OutFormatter[AlignmentRecord] {


From the Jenkins failures, looks like the default ctr parameter value isn't available to Python. Could you add a no-arg apply method or update the Python code?

Resolves bigdatagenomics#1702.

Resolves bigdatagenomics#1703.

AmplabJenkins · 2017-09-12T20:55:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2361/
Test PASSed.

heuermh · 2017-09-12T22:13:05Z

Thank you, @fnothaft

heuermh reviewed Aug 30, 2017

View reviewed changes

heuermh modified the milestone: 0.23.0 Sep 1, 2017

fnothaft force-pushed the issues/1701-1702-ifq-cleanup branch from 69dcfa5 to 9a08d2c Compare September 6, 2017 07:19

fnothaft force-pushed the issues/1701-1702-ifq-cleanup branch from 9a08d2c to c8550bc Compare September 6, 2017 07:46

heuermh requested changes Sep 6, 2017

View reviewed changes

fnothaft added 2 commits September 12, 2017 13:31

[ADAM-1702] Validate read numbers in InterleavedFASTQInFormatter.

2a1e13a

Resolves bigdatagenomics#1702.

[ADAM-1703] Expose ValidationStringency in AnySAMOutFormatter.

f386347

Resolves bigdatagenomics#1703.

fnothaft force-pushed the issues/1701-1702-ifq-cleanup branch from c8550bc to f386347 Compare September 12, 2017 20:32

heuermh approved these changes Sep 12, 2017

View reviewed changes

heuermh merged commit 9cb0c76 into bigdatagenomics:master Sep 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Misc cleanup needed for bigdatagenomics/cannoli#65 #1704

Misc cleanup needed for bigdatagenomics/cannoli#65 #1704

fnothaft commented Aug 30, 2017

heuermh Aug 30, 2017

fnothaft Aug 30, 2017

heuermh Aug 30, 2017

fnothaft Sep 6, 2017

coveralls commented Aug 30, 2017 •

edited

Loading

AmplabJenkins commented Aug 30, 2017

fnothaft commented Sep 6, 2017

AmplabJenkins commented Sep 6, 2017

coveralls commented Sep 6, 2017 •

edited

Loading

coveralls commented Sep 6, 2017

AmplabJenkins commented Sep 6, 2017

heuermh Sep 6, 2017

AmplabJenkins commented Sep 12, 2017

heuermh commented Sep 12, 2017

Misc cleanup needed for bigdatagenomics/cannoli#65 #1704

Misc cleanup needed for bigdatagenomics/cannoli#65 #1704

Conversation

fnothaft commented Aug 30, 2017

heuermh Aug 30, 2017

Choose a reason for hiding this comment

fnothaft Aug 30, 2017

Choose a reason for hiding this comment

heuermh Aug 30, 2017

Choose a reason for hiding this comment

fnothaft Sep 6, 2017

Choose a reason for hiding this comment

coveralls commented Aug 30, 2017 • edited Loading

AmplabJenkins commented Aug 30, 2017

Build result: FAILURE

fnothaft commented Sep 6, 2017

AmplabJenkins commented Sep 6, 2017

Build result: FAILURE

coveralls commented Sep 6, 2017 • edited Loading

coveralls commented Sep 6, 2017

AmplabJenkins commented Sep 6, 2017

Build result: FAILURE

heuermh Sep 6, 2017

Choose a reason for hiding this comment

AmplabJenkins commented Sep 12, 2017

heuermh commented Sep 12, 2017

coveralls commented Aug 30, 2017 •

edited

Loading

coveralls commented Sep 6, 2017 •

edited

Loading