20240122 developer call notes

[hammer]

Pre-notes

• Notes from previous meeting

PRs

• sgkit-publication PR#94 Sequential VCF parsing

Issues

• Yan Wong issues
• sgkit#1168 VCF parsing memory usage
• sgkit#1169 CSI issues
• sgkit-publication#78 Statistical genetics section
• bed-reader#22 read from cloud storage
• sgkit-publication#92 SCVR

Discussions

• Experimental sequential VCF parsing (paper repo)
• numpy#24300 NumPy 2.0

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Jerome Kelleher
4. Ben Jeffery
5. Julian Stamp
6. Tim Millar

Discussion

• Tim’s talk
  • Audience of 40 - 50 ppl
  • DivBrowse: variant browser built on top of scikit-allel to move to sgkit
  • DivBrowse author waiting to hear on stability of data structure
  • Would we consider splitting out the core data structure?
  • Maybe highlight Zarr VCF spec more directly?
• Julian intro
  • Could use some help with Dask
  • Tim notes he felt his audience would stumble on Dask too
  • Ben notes it’s hard to size the Dask cluster correctly
• UKB VCFs to Zarr issues part 1
  • Experimenting locally to get chunk sizes correct
  • Still having issues when moving to cluster
  • Multiple TBs per chromosome
  • Header for each part lists all contigs for all parts
  • JK: post CSI file to help debug partition_into_regions fails to partition correctly in when absent contigs are listed in the header. #1169
  • TW: focused on TBI when we wrote our code
  • JK: most things are CSI now
• Yan Wong issues
  • Documenting sgkit-compatible tsinfer
• sgkit-publication#92 SCVR
  • JH: do we need to evolve our Zarr VCF spec to accommodate these strategies?
  • JK: future work!
  • TW: local alleles will address the PL issue
  • TW: not sure about interactions with chunking
• sgkit-publication#78
  • 2 paths: scale up and scale out
  • Scale up worked for chr20 when bypassing QC and just doing regression
  • Scale up issue for users: quota requests for high memory machines denied!
  • Scale up next step: make full HAPNEST manhattan plot with bigger boxes
  • Scale out died on Coiled due to Numba cache issue possibly caused by Docker
  • Scale out next step: try to run on Dask cloud provider VMs
• UKB VCFs to Zarr issues part 2
  • Tried single partition: removing fields, changing parameters doesn’t get there
  • JK: wrote sequential converter (sgkit-publication PR#94) that doesn’t use Dask, just threads and processes
  • JK: sequential converter is running, will take 60 - 100 hours for biggest partition
  • JK: package with or separately from sgkit?
  • TW: does it use Numba tricks we used in VCF writing?
  • JK: trick is to write chunk-aligned values and never rechunk
  • TW: you know the size of each partition? JK: added code to cyvcf2!
  • JH: we have an issue for alternative execution engines for VCF conversion sgkit#1130
  • JH: this code should be in the VCF module, not as opinionated about the function name
  • JH: only input about API design is that a single conversion function that calls to read then write is hard to debug
  • TW: JK’s implementation is a single function!