address scaling limitation

[sreichl]

When data gets too large the conversion from sparse matrix format to dense matrices crashes.
Here with 120k cells and 22k features (already reduced from 28k).

Warning messages:
1: In .sparse2dense(e1) :
  sparse->dense coercion: allocating vector of size 20.6 GiB
2: In .sparse2dense(e2) :
  sparse->dense coercion: allocating vector of size 20.6 GiB
....
Processing rep1
Error in nt_data %*% model.matrix : 
  Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Calls: CalcPerturbSig -> PerturbDiff -> %*% -> %*%

ideas

• reduce data size further by dim. red.
• split the data per KO after PCA and perform the steps per KO -> aggregate data before LDA
  • Question: do the steps work per KO or require all the data?

[sreichl]

• check how much memory R is actually getting
• reimplement using scanpy's pertpy version of Mixscape: https://pertpy.readthedocs.io/en/latest/usage/tools/pertpy.tools.Mixscape.html
• DR: PCA didn't work when re-installed Seurat. Fix: older version. -> could be related?
• DR: reinstall Matrix package with version: Seurat 4.4.0, Matrix 1.6.5, irlba 2.3.5.1

[sreichl]

• read out all package versions and put in env.yaml if that using Seurat 4.4. worked

[sreichl]

satijalab/seurat#8100
https://support.bioconductor.org/p/9155132/#9155137
https://stackoverflow.com/questions/77442906/runumap-code-error-after-merged-spatial-transcriptomic-objects
bwlewis/irlba#70
https://stat.ethz.ch/pipermail/r-package-devel/2023q4/010051.html
jlmelville/uwot#115