Vcf work rdd master merge #124
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adding ability to convert reference FASTA files for nucleotide sequences
Add initial documentation on contributing
…mmand. This commit fixes issue 92 (#92). The old style of encoding the "optional fields" from the SAM/BAM was to store them as key=value pairs in the ADAMRecord.attributes string. However, this loses information about the _type_ of the tag/value, which is necessary if we want to reconstruct the original value type (for example, for re-exporting BAM files from ADAM files). This update is non-backwards-compatible, changing the format of the attributes field to tag:type:value and introducing a new Attribute class for parsing and handling these values. It also adds functions to AdamRDDFunctions to allow for filtering and subsetting of reads based on their tags, or to count the number of distinct tags or tag-values across a set of reads.
Encoding tag types in the ADAMRecord attributes, adding the 'tags' command
Cleaning up change documentation.
We've been getting intermittent errors, with respect to Spark being unable to bind to a port, in the context of repeated unit tests. This apparently is a known problem, see the thread here: http://blog.quantifind.com/posts/spark-unit-test/ and the follow-up from Matei here: https://groups.google.com/forum/#!topic/spark-users/MeVzgoJXm8I The upshot is that we need to clearProperty('spark.driver.port') when we shut down our sparkContext after a sparkTest.
Added the port erasure to SparkFunSuite's cleanup.
- A plugin class can be defined outside of the Adam jar, but run through the normal AdamMain - An example plugin, the "Take10Plugin" is included in the test directory - Adds a test suite to the cli module, which can reference the items available in the core module - Adds notion of AccessControl to control the records which can be accessed
- Functional test - More comments
Adding new PluginExecutor command
… vcf-work-rdd-master-merge Conflicts: adam-cli/src/main/scala/edu/berkeley/cs/amplab/adam/cli/Adam2Vcf.scala adam-core/src/main/scala/edu/berkeley/cs/amplab/adam/converters/VariantContextConverter.scala adam-core/src/main/scala/edu/berkeley/cs/amplab/adam/models/ADAMVariantContext.scala adam-core/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamContext.scala adam-core/src/main/scala/edu/berkeley/cs/amplab/adam/rdd/AdamRDDFunctions.scala adam-core/src/test/scala/edu/berkeley/cs/amplab/adam/converters/VariantContextConverterSuite.scala adam-core/src/test/scala/edu/berkeley/cs/amplab/adam/models/ADAMVariantContextSuite.scala adam-format/src/main/resources/avro/adam.avdl
|
One or more automated tests failed |
|
Can we file an issue to improve the filter representation after the topic branch is merged into master? |
|
One or more automated tests failed |
|
@nealsid The tests are failing Is that an expected failure? I know that some of the intermediate work may not pass all tests. |
|
I think it make sense to fix the build now, since it's close to being ready to be merged back to master, so I'll ping the pull request when it's green and ready for merging into vcf-work |
|
Still needs work, unfortunately. Merging ReferenceRegion the new Variant schema took a little work, and the tests still aren't passing. |
|
One or more automated tests failed |
… and left outer join bug) Update code to use integers for reference/contig ids
|
Jenkins, test this please |
|
All automated tests passed. |
|
Nice! All the tests pass now. Thanks, Neal! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a merge of master into the vcf-work branch. It contains all the changes of vcf-work-rdd, so it has more than necessary to be reviewed and should drop in size once vcf-work-rdd is merged into vcf-work.