[help] despite tar_group tar_read(..., branches = 1) load the entire object

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a block as such

  tarchetypes::tar_group_by(
        metadata_grouped_pseudobulk,
        metadata_split_pseudobulk_SMALL |> bind_rows(metadata_split_pseudobulk_BIG),
        chunk,
        resources = tar_resources(crew = tar_resources_crew("slurm_1_150"))
      ) 

But when I do

tar_read(metadata_grouped_pseudobulk, branches = 1)

I get the whole object loaded (45Gb)

I tried to load the object and this is a little more details.

> tar_meta(metadata_grouped_pseudobulk, store = glue("{result_directory}/_targets"))
# A tibble: 1 × 18
  name         type  data  command depend   seed path  time                size    bytes format repository iteration parent children seconds warnings error
  <chr>        <chr> <chr> <chr>   <chr>   <int> <lis> <dttm>              <chr>   <dbl> <chr>  <chr>      <chr>     <chr>  <list>     <dbl> <chr>    <chr>
1 metadata_gr… stem  e4f5… a2f9ad… 38a9a… 1.87e8 <chr> 2024-11-22 18:32:43 2970… 1.85e10 qs     local      group     NA     <chr>       200. NA       NA   

So this target is group

> x = tar_read(metadata_grouped_pseudobulk, store = glue("{result_directory}/_targets"), branches = 1)

Not sure why branches = 1 is not respected and the whole thing is loaded

> x

# A tibble: 30,150 × 4
   chunk                                     data            number_of_cells tar_group
   <chr>                                     <list>                    <int>     <int>
 1 0001ab628e84b4ac58fade7b8cebf0b2___1.h5ad <SnglCllE[,4]>             1324         1
 2 000623ab84de100ff7477a7895714985___1.h5ad <SnglCllE[,28]>             906         2
 3 0007a5e25d8d0d8ab26504708cf8e370___1.h5ad <SnglCllE[,36]>            9053         3
 4 00080046929842e7453c960ef959b345___1.h5ad <SnglCllE[,2]>              106         4
 5 00086827a1937ff2fd593de2fc11cfc5___1.h5ad <SnglCllE[,1]>             2471         5
 6 0008803ce8cd9c3a93bc249240ca0ce4___1.h5ad <SnglCllE[,45]>           15230         6
 7 0008ce45a76b37ed3cb51ee74080b9fd___1.h5ad <SnglCllE[,2]>                2         7
 8 000de73f750235d64e02d7ca9e94a414___1.h5ad <SnglCllE[,4]>             1892         8
 9 000e85e92149192a77fc16aeab485ed8___1.h5ad <SnglCllE[,5]>              104         9
10 0014aa5a36d997388157f0ef22d22f14___1.h5ad <SnglCllE[,7]>              200        10
# ℹ 30,140 more rows
# ℹ Use `print(n = ...)` to see more rows

x |> attributes()
$row.names
   [1]    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29
  [30]   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58
  [59]   59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87
  [88]   88   89   90   91   92   93   94   95   96   97   98   99  100  101  102  103  104  105  106  107  108  109  110  111  112  113  114  115  116
 [117]  117  118  119  120  121  122  123  124  125  126  127  128  129  130  131  132  133  134  135  136  137  138  139  140  141  142  143  144  145
 [146]  146  147  148  149  150  151  152  153  154  155  156  157  158  159  160  161  162  163  164  165  166  167  168  169  170  171  172  173  174
 [175]  175  176  177  178  179  180  181  182  183  184  185  186  187  188  189  190  191  192  193  194  195  196  197  198  199  200  201  202  203
 [204]  204  205  206  207  208  209  210  211  212  213  214  215  216  217  218  219  220  221  222  223  224  225  226  227  228  229  230  231  232
 [233]  233  234  235  236  237  238  239  240  241  242  243  244  245  246  247  248  249  250  251  252  253  254  255  256  257  258  259  260  261
 [262]  262  263  264  265  266  267  268  269  270  271  272  273  274  275  276  277  278  279  280  281  282  283  284  285  286  287  288  289  290
 [291]  291  292  293  294  295  296  297  298  299  300  301  302  303  304  305  306  307  308  309  310  311  312  313  314  315  316  317  318  319
 [320]  320  321  322  323  324  325  326  327  328  329  330  331  332  333  334  335  336  337  338  339  340  341  342  343  344  345  346  347  348
 [349]  349  350  351  352  353  354  355  356  357  358  359  360  361  362  363  364  365  366  367  368  369  370  371  372  373  374  375  376  377
 [378]  378  379  380  381  382  383  384  385  386  387  388  389  390  391  392  393  394  395  396  397  398  399  400  401  402  403  404  405  406
 [407]  407  408  409  410  411  412  413  414  415  416  417  418  419  420  421  422  423  424  425  426  427  428  429  430  431  432  433  434  435
 [436]  436  437  438  439  440  441  442  443  444  445  446  447  448  449  450  451  452  453  454  455  456  457  458  459  460  461  462  463  464
 [465]  465  466  467  468  469  470  471  472  473  474  475  476  477  478  479  480  481  482  483  484  485  486  487  488  489  490  491  492  493
 [494]  494  495  496  497  498  499  500  501  502  503  504  505  506  507  508  509  510  511  512  513  514  515  516  517  518  519  520  521  522
 [523]  523  524  525  526  527  528  529  530  531  532  533  534  535  536  537  538  539  540  541  542  543  544  545  546  547  548  549  550  551
 [552]  552  553  554  555  556  557  558  559  560  561  562  563  564  565  566  567  568  569  570  571  572  573  574  575  576  577  578  579  580
 [581]  581  582  583  584  585  586  587  588  589  590  591  592  593  594  595  596  597  598  599  600  601  602  603  604  605  606  607  608  609
 [610]  610  611  612  613  614  615  616  617  618  619  620  621  622  623  624  625  626  627  628  629  630  631  632  633  634  635  636  637  638
 [639]  639  640  641  642  643  644  645  646  647  648  649  650  651  652  653  654  655  656  657  658  659  660  661  662  663  664  665  666  667
 [668]  668  669  670  671  672  673  674  675  676  677  678  679  680  681  682  683  684  685  686  687  688  689  690  691  692  693  694  695  696
 [697]  697  698  699  700  701  702  703  704  705  706  707  708  709  710  711  712  713  714  715  716  717  718  719  720  721  722  723  724  725
 [726]  726  727  728  729  730  731  732  733  734  735  736  737  738  739  740  741  742  743  744  745  746  747  748  749  750  751  752  753  754
 [755]  755  756  757  758  759  760  761  762  763  764  765  766  767  768  769  770  771  772  773  774  775  776  777  778  779  780  781  782  783
 [784]  784  785  786  787  788  789  790  791  792  793  794  795  796  797  798  799  800  801  802  803  804  805  806  807  808  809  810  811  812
 [813]  813  814  815  816  817  818  819  820  821  822  823  824  825  826  827  828  829  830  831  832  833  834  835  836  837  838  839  840  841
 [842]  842  843  844  845  846  847  848  849  850  851  852  853  854  855  856  857  858  859  860  861  862  863  864  865  866  867  868  869  870
 [871]  871  872  873  874  875  876  877  878  879  880  881  882  883  884  885  886  887  888  889  890  891  892  893  894  895  896  897  898  899
 [900]  900  901  902  903  904  905  906  907  908  909  910  911  912  913  914  915  916  917  918  919  920  921  922  923  924  925  926  927  928
 [929]  929  930  931  932  933  934  935  936  937  938  939  940  941  942  943  944  945  946  947  948  949  950  951  952  953  954  955  956  957
 [958]  958  959  960  961  962  963  964  965  966  967  968  969  970  971  972  973  974  975  976  977  978  979  980  981  982  983  984  985  986
 [987]  987  988  989  990  991  992  993  994  995  996  997  998  999 1000
 [ reached getOption("max.print") -- omitted 29150 entries ]

$names
[1] "chunk"           "data"            "number_of_cells" "tar_group"      

$class
[1] "tbl_df"     "tbl"        "data.frame"

sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Red Hat Enterprise Linux 9.4 (Plow)

Matrix products: default
BLAS:   /stornext/System/data/software/rhel/9/base/tools/R/4.4.1/lib64/R/lib/libRblas.so 
LAPACK: /stornext/System/data/software/rhel/9/base/tools/R/4.4.1/lib64/R/lib/libRlapack.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Australia/Melbourne
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] SingleCellExperiment_1.26.0 SummarizedExperiment_1.34.0 Biobase_2.64.0              GenomicRanges_1.56.1        GenomeInfoDb_1.40.1        
 [6] IRanges_2.38.1              S4Vectors_0.42.1            BiocGenerics_0.50.0         MatrixGenerics_1.16.0       matrixStats_1.4.1          
[11] duckdb_1.0.0-2              DBI_1.2.3                   arrow_17.0.0.1              glue_1.8.0                  targets_1.9.0.9000         
[16] lubridate_1.9.3             forcats_1.0.0               stringr_1.5.1               dplyr_1.1.4                 purrr_1.0.2                
[21] readr_2.1.5                 tidyr_1.3.1                 tibble_3.2.1                ggplot2_3.5.1               tidyverse_2.0.0            

loaded via a namespace (and not attached):
 [1] tidyselect_1.2.1        stringfish_0.16.0       digest_0.6.37           base64url_1.4           timechange_0.3.0        lifecycle_1.0.4        
 [7] secretbase_1.0.3        processx_3.8.4          magrittr_2.0.3          compiler_4.4.1          rlang_1.1.4             tools_4.4.1            
[13] igraph_2.1.1            utf8_1.2.4              yaml_2.3.10             data.table_1.16.2       knitr_1.48              S4Arrays_1.4.1         
[19] bit_4.5.0               DelayedArray_0.30.1     abind_1.4-8             HDF5Array_1.32.1        withr_3.0.2             grid_4.4.1             
[25] fansi_1.0.6             job_0.3.1               colorspace_2.1-1        Rhdf5lib_1.26.0         scales_1.3.0            cli_3.6.3              
[31] crayon_1.5.3            generics_0.1.3          RcppParallel_5.1.9      rstudioapi_0.16.0       httr_1.4.7              tzdb_0.4.0             
[37] rhdf5_2.48.0            zlibbioc_1.50.0         assertthat_0.2.1        XVector_0.44.0          vctrs_0.6.5             Matrix_1.7-0           
[43] jsonlite_1.8.9          callr_3.7.6             hms_1.1.3               bit64_4.5.2             codetools_0.2-20        ps_1.8.1               
[49] stringi_1.8.4           gtable_0.3.6            UCSC.utils_1.0.0        munsell_0.5.1           pillar_1.9.0            rhdf5filters_1.16.0    
[55] GenomeInfoDbData_1.2.12 R6_2.5.1                lattice_0.22-6          backports_1.5.0         Rcpp_1.0.13-1           SparseArray_1.4.8      
[61] qs2_0.1.1               xfun_0.48               pkgconfig_2.0.3        

And if I debug the target that uses metadata_grouped_pseudobulk, it loads for 10 minutes 46Gb before returning just one line. It is like the whole target is loaded into memory before being filtered.

The pipeline is complex to reproduce here, but I hope this would ring some bell.

---

On a related side

Let's try to solve this changing format

tar_target(
        metadata_grouped_pseudobulk,
        metadata_split_pseudobulk_SMALL |> 
          bind_rows(metadata_split_pseudobulk_BIG) |> 
          group_by(chunk) |> 
          tar_group(),
        iteration = "group",
        resources = tar_resources(crew = tar_resources_crew("slurm_1_150"))
      )

I get the error

Last error message ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    iteration = "group" requires data frame targets.

But if I re-execute with debug

debug = "metadata_grouped_pseudobulk",

and

callr_function = NULL

I have this

metadata_split_pseudobulk_SMALL
# A tibble: 30,141 × 4
   chunk                                     data            number_of_cells tar_group
   <chr>                                     <list>                    <int>     <int>
 1 0001ab628e84b4ac58fade7b8cebf0b2___1.h5ad <SnglCllE[,4]>             1324         2
 2 000623ab84de100ff7477a7895714985___1.h5ad <SnglCllE[,28]>             906         3
 3 0007a5e25d8d0d8ab26504708cf8e370___1.h5ad <SnglCllE[,36]>            9053         4
 4 00080046929842e7453c960ef959b345___1.h5ad <SnglCllE[,2]>              106         5
 5 00086827a1937ff2fd593de2fc11cfc5___1.h5ad <SnglCllE[,1]>             2471         6
 6 0008803ce8cd9c3a93bc249240ca0ce4___1.h5ad <SnglCllE[,45]>           15230         7
 7 0008ce45a76b37ed3cb51ee74080b9fd___1.h5ad <SnglCllE[,2]>                2         8
 8 000de73f750235d64e02d7ca9e94a414___1.h5ad <SnglCllE[,4]>             1892         9
 9 000e85e92149192a77fc16aeab485ed8___1.h5ad <SnglCllE[,5]>              104        10
10 0014aa5a36d997388157f0ef22d22f14___1.h5ad <SnglCllE[,7]>              200        11
# ℹ 30,131 more rows

 metadata_split_pseudobulk_BIG
# A tibble: 9 × 4
  chunk                                     data            number_of_cells tar_group
  <chr>                                     <list>                    <int>     <int>
1 015c36c90414f852801acb521bb86365___3.h5ad <SnglCllE[,1]>            55078         1
2 23c787d5ee7a4038b637bd859c1dfe61___3.h5ad <SnglCllE[,9]>            50256         2
3 47fa29e316b0f50d12602a97565b7a9e___5.h5ad <SnglCllE[,1]>            89020         3
4 68195aad60a4c600acc8678093d5cb3a___3.h5ad <SnglCllE[,7]>            51137         4
5 8aacbdf9a09696b54bf1c415f158de76___4.h5ad <SnglCllE[,1]>            69380         5
6 ba3c2bd0378e03705974110af5584433___4.h5ad <SnglCllE[,1]>            76621         6
7 ed95f223c29a9364194c1cf58933f68b___5.h5ad <SnglCllE[,1]>            89583         7
8 f1d00be606e0bad4055dbf005cb542a5___4.h5ad <SnglCllE[,12]>           62509         8
9 f2bf9c28ab5262b1d50de88eca0f3e15___3.h5ad <SnglCllE[,5]>            54345         9

 metadata_split_pseudobulk_SMALL |> 
+           bind_rows(metadata_split_pseudobulk_BIG) |> 
+           group_by(chunk) |> 
+           tar_group()
# A tibble: 30,150 × 4
   chunk                                     data            number_of_cells tar_group
   <chr>                                     <list>                    <int>     <int>
 1 0001ab628e84b4ac58fade7b8cebf0b2___1.h5ad <SnglCllE[,4]>             1324         1
 2 000623ab84de100ff7477a7895714985___1.h5ad <SnglCllE[,28]>             906         2
 3 0007a5e25d8d0d8ab26504708cf8e370___1.h5ad <SnglCllE[,36]>            9053         3
 4 00080046929842e7453c960ef959b345___1.h5ad <SnglCllE[,2]>              106         4
 5 00086827a1937ff2fd593de2fc11cfc5___1.h5ad <SnglCllE[,1]>             2471         5
 6 0008803ce8cd9c3a93bc249240ca0ce4___1.h5ad <SnglCllE[,45]>           15230         6
 7 0008ce45a76b37ed3cb51ee74080b9fd___1.h5ad <SnglCllE[,2]>                2         7
 8 000de73f750235d64e02d7ca9e94a414___1.h5ad <SnglCllE[,4]>             1892         8
 9 000e85e92149192a77fc16aeab485ed8___1.h5ad <SnglCllE[,5]>              104         9
10 0014aa5a36d997388157f0ef22d22f14___1.h5ad <SnglCllE[,7]>              200        10
# ℹ 30,140 more rows
# ℹ Use `print(n = ...)` to see more rows

and pressing

c

everything works fine without error, but still the object cannot be loaded by group. Anything to do with qs2 saving?

[wlandau]

Just as dplyr::group_by() only declares the groups for downstream processing, tarchetypes::tar_group_by() only declares the groups for downstream dynamic branching. In other words, a tar_group_by() target itself is not branched, but a downstream target with pattern = map(metadata_grouped_pseudobulk) will run one dynamic branch for every group of metadata_grouped_pseudobulk. In tar_read(), the branches argument is irrelevant if the target does not have dynamic branches.

[stemangiola]

I see.

The ability to create a pointer would be great to represent the merged list or data frames but does not actually create a monolithic one, losing branching, as the merged list is too heavy to be loaded in it's entirety.

In other words A_light and A_heavy are targets for branched list/data frames.

I want to consolidate them to avoid duplicating code if I don't need it and create a virtual target A that points to A_light and A_heavy so I can do

tar_pointer(A, c(A_light, A_heavy)) # A is not a real target but an indication to iterate over multiple targets as if they were one.
tar_targets(B, A |> do_something_branched())

This I this would be a great addition, again to allow to apply targets to real-world large-scale pipelines.

[wlandau]

The ability to create a pointer would be great to represent the merged list or data frames but does not actually create a monolithic one, losing branching, as the merged list is too heavy to be loaded in it's entirety.

targets automatically creates the merged list when you mention the name of the target in a downstream one without branching over it with pattern = .... In your case, you can avoid this with:

list(
  tar_target(A_light_branched, do_something_branched(A_light), pattern = map(A_light)),
  tar_target(A_heavy_branched, do_something_branched(A_heavy), pattern = map(A_heavy))
)

If do_something_branched() returns light objects and the downstream part of the pipeline does not mention A_heavy, then aggregation by targets should not be burdensome.