Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transition to SQLite backend - use transcript-specific databases #192

Open
wants to merge 100 commits into
base: new_data_store
Choose a base branch
from
Open

Transition to SQLite backend - use transcript-specific databases #192

wants to merge 100 commits into from

Conversation

hendrikweisser
Copy link

Supersedes #184.

This implementation uses one separate SQLite database for each transcript, plus one "master" database, to ease parallelisation. All databases are created during the "eventalign_collapse" step, then extended during the "sampcomp" step.

@hendrikweisser
DataStore: small fix in SQL query, add to-do comment
78e923b
@hendrikweisser
Eventalign_collapse: add flag for writing to DB (or TSV file), fix er…
e5d7621 
…ror in DataStore call ('add_read' renamed to 'store_read')
@hendrikweisser
Merge branch 'new_data_store' of https://github.com/tleonardi/nanocom…
51a87e4 
…pore into new_data_store
@hendrikweisser
better log message in 'DataStore.__init__'
d7edde9
@hendrikweisser
Merge branch 'new_data_store' of https://github.com/tleonardi/nanocom…
ba2da54 
…pore into new_data_store
@hendrikweisser
coding style (added spaces for readability)
afe005c
@hendrikweisser
store read-level kmer stats in database (needed for whitelisting)
7f60da1
@hendrikweisser
Whitelist: read data from SQLite, filter reads during query
da7eb4b
@hendrikweisser
add function to check validity of sample dictionary to 'common.py'
2fefe7c
@hendrikweisser
new class 'DatabaseWrapper' for reusable DB interaction code; adapt '…
8f52c1c 
…Whitelist' accordingly
@hendrikweisser
Whitelist: fix filtering condition for sample subsets
6121376
@hendrikweisser
consolidate database code in 'DataStore', remove 'DatabaseWrapper' (n…
989af13 
…ot yet tested)
@hendrikweisser
DataStore: small fixes, move 'DBCreateMode' (enum) to top level
6dbbdb2
@hendrikweisser
Eventalign_collapse: small fix in 'DataStore' call
031a995
@hendrikweisser
Whitelist: updated 'DataStore' call
9cc0806
@hendrikweisser
SampComp: get data from SQLite DB; some refactoring
a1cb349
@hendrikweisser
TxComp: update to changes in 'SampComp'; some refactoring
fd0d446
@hendrikweisser
refactor DataStore, create child classes 'DS_EventAlign' and 'DS_Samp…
69563e0 
…Comp'
@hendrikweisser
use 'DataStore_EventAlign' in 'Eventalign_collapse' and 'Whitelist'
b1ad899
@hendrikweisser
TxComp: simplify 'txCompare' results data structure (remove 'lowCov' …
b8de4e9 
…elements)
@hendrikweisser
SampComp: small logging fix and update of 'txCompare' results
7fae6d8
@hendrikweisser
DataStore: add functions to store sample information and whitelisted …
9e96e8a 
…reads
@hendrikweisser
SampComp: store whitelisted reads in SQLite DB
63429d5
@hendrikweisser
TxComp: use 'ST' as abbrev. for (Student) t-test
edf0081
@hendrikweisser
DataStore: fix DB init (tables creation), rework DB schema for SampCo…
536ab64 
…mp results
@hendrikweisser
SampComp: remove unused parameters, write output to SQLite
24881a4
@hendrikweisser
TxComp: coding style - added whitespace
fa0ec95
@hendrikweisser
DataStore: split 'gmm_results' SQL table into two; improve error logg…
112e50d 
…ing during table creation
@hendrikweisser
add new class 'PostProcess' for data export etc. (work in progress)
6d00f2e
@hendrikweisser
SampComp/TxComp/DataStore: limit to one univariate and one GMM-based …
d8324c7 
…statistical test per k-mer
@hendrikweisser
DataStore/SampComp: add functions to reset databases to pre-SampComp …
b2bca8a 
…state
@hendrikweisser
SampComp/DataStore: option to store fitted GMMs in transcript databases
cf9a1e0
@hendrikweisser
TxComp: don't scale intensities/dwell times before GMM fit
ea6e720
@hendrikweisser
SampComp/Whitelist: perform downsampling of reads per-sample (now as …
56dff1c 
…part of filtering in 'Whitelist')
@hendrikweisser
DataStore: calculate standard-scaled kmer intensities/dwell times and…
23fbe23 
… store them in transcript DB (warning: slow)
@hendrikweisser
Eventalign_collapse: calculate mean and std. dev. of kmer intensities…
3665ba3 
…/dwell times per read (to allow data scaling)
@hendrikweisser
DataStore: store read summary stats for data scaling; remove scaling …
0672b0e 
…calculations; use 'dwelltime' consistently in DB column names
@hendrikweisser
TxComp: use 'dwelltime' consistently
e1f6083
@hendrikweisser
SampComp: scale intensity/dwell time data when reading from DB; make …
6613d75 
…significance filtering optional; use 'dwelltime' consistently
@hendrikweisser
Eventalign_collapse: add option to filter transcripts by accession
cebf5c7
@hendrikweisser
SampComp/Eventalign_collapse: small fixes
8ed34cc
@hendrikweisser
store log10-transformed dwell times in transcript databases
f49ae81
@hendrikweisser
TxComp: return all fitted GMMs, not just best
1897a82
@hendrikweisser
SampComp/DataStore: add option to store components of all fitted GMMs…
b93b15a 
… in database
@hendrikweisser
TxComp/DataStore: store more information about fitted GMMs
9e464b2
@hendrikweisser
SampComp/DataStore: use per-sequence mean/std.dev. for intensity scal…
03af595 
…ing; store scaling parameters in DB
@hendrikweisser
SampComp: stop pre-scaling intensities; TxComp: scale data for each t…
6f35500 
…ranscript position before GMM fit; DataStore: reverse scaling before writing GMM parameters to DB
@hendrikweisser
allow running 'eventalign_collapse' on a single input (via file or st…
4833de6 
…din)
@hendrikweisser
DataStore: fix constraint failure when 'eventalign_collapse' is used …
7d0b502 
…multiple times with input via stdin
@hendrikweisser
Eventalign_collapse: improved log message
3c24c16
@hendrikweisser
main: fix error in argument parsing for 'eventalign_collapse'
d7253d7
@hendrikweisser
TxComp: for 2-GMMs, ensure component 1 has lower intensity than compo…
7d78f4d 
…nent 2 (swap them if necessary)
@hendrikweisser
main: small fixes, doc improvements
7945dcf
@hendrikweisser
Eventalign_collapse: fix std. dev. calculations for reads with only o…
2f6758f 
…ne kmer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@a-slide a-slide Awaiting requested review from a-slide a-slide is a code owner

@tleonardi tleonardi Awaiting requested review from tleonardi tleonardi is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@hendrikweisser