20240902 developer call notes

[tomwhite]

20240902

Pre-notes

• Notes from previous meeting

PRs

Issues

• Tensorstore is surprisingly slow vcf-zarr-publication#161
• Run test_count_call_alleles on Cubed Run test_count_call_alleles on Cubed #1249

Discussions

• Paper reviews are in - very positive, overall

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Will Tyler
4. Jerome Kelleher
5. Ben Jeffery

Discussion

• Paper
  • JH: add something about vcztools?
  • JK: mention it’s an early prototype, good decoding speed, leave door open for later paper.
  • JK: that will address reviewer 2 comments
  • JK: contact Augusto Rendon to talk about S3 dataset [show vcztools faster than bcftools to decode on some of their data], also a 1K genome analysis?
  • JH: timeline?
  • JK: wait on AR, then about a week
  • JH: PGEN mention: https://github.com/chrchang/plink-ng/blob/master/pgen_spec/pgen_spec.pdf
  • JK: just for GT fields - not a big space saving
• Scaling sgkit/linalg
  • TW: spoke to Eric C, who made the point that JAX is an array API, but Numba is more general. So are there examples that we can’t express with JAX - e.g. HWE?
  • JK: TS is a bit of a disappointment (using C++).
  • JH: Eric Betzig is generating 1PB of microscopy data a day. Matlab framework with C++ to analyse Zarr. https://github.com/abcucberkeley/cpp-zarr
  • JH: Can connect with TS team too
  • JK: Z5 another option, but possibly not so active
• Run test_count_call_alleles on Cubed Run test_count_call_alleles on Cubed #1249
  • TW: there’s a Dask regression.
  • JK: could have a common function to dispatch to da.map_blocks or Cubed (via xarray)
• Vcztools
  • JK: Should we do a release?
  • TW: Yes!
  • JK: what would you like to work on WT?
  • WT: knocking out issues, also C++ work
  • JK: SAIGE might be good - will send an email