speed ups to rna quant ingest #1564
Conversation
|
Added fix to oidc problem also to close #1566. |
Codecov Report
@@ Coverage Diff @@
## master #1564 +/- ##
==========================================
+ Coverage 84.62% 85.61% +0.99%
==========================================
Files 33 33
Lines 7166 7188 +22
Branches 897 898 +1
==========================================
+ Hits 6064 6154 +90
+ Misses 937 851 -86
- Partials 165 183 +18
Continue to review full report at Codecov.
|
|
@david4096 I removed uuid id generation, which was very slow. The auto increment was just for convenience. The ids can be generated by incrementing a counter in the code, too. On reflection, this will need to be changed to assign ids incrementally by RnaQunatifiction in order to have reproducible ids. The primary key will need to be changed to be the id column and the rna_quantification_id. |
|
I changed the id as described above and added an ingest test to increase the code coverage. |
| @@ -157,6 +190,7 @@ def writeExpression(self, rnaQuantificationId, quantfilename, | |||
| rawCount, score, units, confidenceLow, | |||
| confidenceHi) | |||
| self._db.addExpression(datafields) | |||
| expressionId += 1 | |||
There was a problem hiding this comment.
Cool! Do you think this same idea will work for adding expressions in parallel (ignoring sqlite)?
There was a problem hiding this comment.
Yes, since each file will be processed within it's own process/thread.
|
|
||
| testTsvFile = os.path.join( | ||
| paths.testDataDir, | ||
| "datasets/dataset1/rnaQuant/rsem_test_data.tsv") |
There was a problem hiding this comment.
This test is really just guaranteeing the whole this doesn't go belly up, which is a great improvement! Also makes me think we could add it to the https://github.com/ga4gh/server/blob/master/scripts/build_test_data.py
* speed ups to rna quant ingest * fixed oidc problem * added tests and changed expression ids to be unique within rnaquant * flake fixes
I made the expression id an auto increment field and sped up reading the expression data from the file. Closes #1549