A more compact and useful representation for unphased data #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, unphased data was stored just like phased data. The only difference was a flag set in the file header, otherwise you still had haploid sample lists that you need to traverse and count the number of copies based on the ploidy. Now we store the number of copies explicitly in the position index, where we reserve 8-bits for this (so supports ploidy up to 255). Each variant row then has an associated number of copies, and you can just skip a variant if it has a number of copies you don't care about. For example, with diploid data you can do analyses with just homozygotes by only scanning variants with numCopies=2, which would massively speed up your analysis.
dcdehaas
commented
Nov 25, 2024
Unphased data can be processed in igdtools, either for conversion, filtering, or just stats. There is also a --force-unphased flag that lets you de-phase an IGD file (usually useful for testing tools, if nothing else). VCF files that are unphased will get converted to unphased IGDs, VCF files that are phased will get converted to phased IGDs, unless --force-unphased is used. VCF files that have mixed phasedness can now be used with picovcf (and igdtools) if forceUnphased (or --force-unphased) are set, and the resulting IGD will be unphased. New test/endtoend/run_tests.py file for non-unit tests, currently only testing igdtools.
Very fast because of the sparse representation. Also run the end-to-end tests at CI time.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, unphased data was stored just like phased data. The only difference was a flag set in the file header, otherwise you still had haploid sample lists that you need to traverse and count the number of copies based on the ploidy.
Now we store the number of copies explicitly in the position index, where we reserve 8-bits for this (so supports ploidy up to 255). Each variant row then has an associated number of copies, and you can just skip a variant if it has a number of copies you don't care about. For example, with diploid data you can do analyses with just homozygotes by only scanning variants with numCopies=2, which would massively speed up your analysis.