MRG: in core, enable downsample within select #2931

bluegenes · 2024-01-17T02:17:07Z

Attempting #1292 in order to move forward sourmash-bio/sourmash_plugin_branchwater#134

Modifies Signature Select to downsample automatically.

for scaled sketches, while checking ksize, we also retain only sketches that have the right scaled or can be downsampled (scaled <= selection.scaled())
next, we iterate through the sketches and downsample any where scaled < selection.scaled()

I thought the change was needed in Signature Select rather than SigStore Select, but lmk if I missed something!

codecov · 2024-01-17T20:25:18Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a0f3016) 86.59% compared to head (1b74d45) 86.61%.

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #2931      +/-   ##
==========================================
+ Coverage   86.59%   86.61%   +0.02%     
==========================================
  Files         135      135              
  Lines       15251    15260       +9     
  Branches     2622     2622              
==========================================
+ Hits        13206    13217      +11     
+ Misses       1745     1743       -2     
  Partials      300      300

Flag	Coverage Δ
hypothesis-py	`25.80% <ø> (ø)`
python	`92.82% <ø> (ø)`
rust	`59.22% <100.00%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bluegenes · 2024-01-22T18:00:16Z

@luizirber @ctb ready for review

ctb

This looks good to me although I'd suggest waiting for Luiz before merging :)

…into select-downsample

luizirber

Nice!

Considering the Select::select function signature:

fn select(mut self, selection: &Selection) -> Result<Self, Error>

I think it is OK to downsample it at the Signature level, and not SigStore

src/core/src/signature.rs

Co-authored-by: Luiz Irber <luizirber@users.noreply.github.com>

bluegenes · 2024-01-23T19:22:58Z

Note that for sourmash_plugin_branchwater compatibility, we need:

byteorder = "1.4.3"
wasm-bindgen = "0.2.89"
once_cell = "1.18.0"

MSRV: 1.65 Changes/additions: * Clean up and refactor `KmerMinHash::merge` in core (#2973) * core: add scaled selection to manifest; add helper functions for collection and sig/sketch usage (#2948) * core: enable downsample within select (#2931) Updates: * Deps: update typed-builder and histogram, bump MSRV to 1.65 (#2858) * Bump tempfile from 3.9.0 to 3.10.0 (#2979) * Bump rkyv from 0.7.43 to 0.7.44 (#2978) * Bump memmap2 from 0.9.3 to 0.9.4 (#2958) * Bump chrono from 0.4.31 to 0.4.33 (#2957) * Bump serde from 1.0.195 to 1.0.196 (#2956) * Bump serde_json from 1.0.111 to 1.0.113 (#2955) * Bump web-sys from 0.3.66 to 0.3.67 (#2939) * Bump wasm-bindgen-test from 0.3.39 to 0.3.40 (#2938) * Bump rayon from 1.8.0 to 1.8.1 (#2937) * Bump ouroboros from 0.18.2 to 0.18.3 (#2936) * Bump histogram from 0.8.4 to 0.9.0 (#2935) * Bump wasm-bindgen from 0.2.89 to 0.2.90 (#2925) * Bump histogram from 0.8.3 to 0.8.4 (#2923) * Bump serde_json from 1.0.110 to 1.0.111 (#2902) * Bump serde from 1.0.194 to 1.0.195 (#2901) * Bump serde_json from 1.0.108 to 1.0.110 (#2896) * Bump ouroboros from 0.18.1 to 0.18.2 (#2894) * Bump tempfile from 3.8.1 to 3.9.0 (#2893) * Bump memmap2 from 0.9.2 to 0.9.3 (#2889) * Bump memmap2 from 0.9.0 to 0.9.2 (#2882) * Bump rkyv from 0.7.42 to 0.7.43 (#2880) * Bump ouroboros from 0.18.0 to 0.18.1 (#2875) * Bump once_cell from 1.18.0 to 1.19.0 (#2874) * Bump rkyv from 0.7.40 to 0.7.42 (#2863) * Bump csv from 1.2.0 to 1.3.0 (#2862) * Bump roaring from 0.10.1 to 0.10.2 (#2865) * Bump web-sys from 0.3.65 to 0.3.66 (#2864) * Bump byteorder from 1.4.3 to 1.5.0 (#2866) * Bump proptest from 1.3.1 to 1.4.0 (#2837)

release notes: https://hackmd.io/SCoVcWS1RhCH-ndQNWF-1A?view # sourmash release v4.8.6 - release notes Minor new features: * re-establish `tax` gather reading flexibility (#2986) * update JOSS paper per pyopensci review (#2964) * Clean up and refactor `KmerMinHash::merge` in core (#2973) * add label output & input options to `compare` and `plot`, for better customization (#2598) * add utilities for using ictv taxonomic ranks with `sourmash tax` (#2608) Bug fixes: * Fix `tax metagenome` to work on gather output created with `--estimate-ani-ci` (#2952) * fix gather memory usage issue by not accumulating `GatherResult` (#2962) * update the CLI docs and help for `search --containment` and `prefetch` (#2971) Documentation updates: * update tutorial to remove bioconda & use sourmash-minimal (#2972) * update readme with maintainers & sourmash comparison info (#2965) * add branchwater reference; make FAQ more visible (#2984) * update FAQ answer on k-mer size (#2899) * update README with repostatus and pyver badges, and Windows support (#2928) * add full column descriptions for `gather` and `prefetch` output (#2954) * add scaled FAQ, adjust ksize answer (#2921) * minor refactoring of gather code, small doc updates (#2953) * Add threshold-bp and scaled relationship to faqs (#2930) Developer updates: * nix updates for pyopensci review (#2975) * add scaled selection to manifest; add helper functions for collection and sig/sketch usage (#2948) * Pre-commit updates (#2427) * fix upload wheel CI (#2974) * release core; bump rust core version to r0.12.1 (#2988) * CI: macos deployment target and maturin updates (#2879) * MRG: bump version to 4.8.6-dev, post-release (#2877) * fix benchmark code & a few other small issues from pyOpenSci review (#2920) * fix uploading of wheels after upload-artifact upgrade. (#2887) * in core, enable downsample within select (#2931) Dependabot updates: * Bump pypa/cibuildwheel from 2.16.4 to 2.16.5 (#2981) * Bump tempfile from 3.9.0 to 3.10.0 (#2979) * Bump rkyv from 0.7.43 to 0.7.44 (#2978) * Bump actions/cache from 3 to 4 (#2933) * Bump actions/download-artifact from 3 to 4 (#2884) * Bump actions/upload-artifact from 3 to 4 (#2883) * Bump cachix/cachix-action from 13 to 14 (#2926) * Bump cachix/install-nix-action from 24 to 25 (#2927) * Bump chrono from 0.4.31 to 0.4.33 (#2957) * Bump getrandom from 0.2.11 to 0.2.12 (#2924) * Bump histogram from 0.8.3 to 0.8.4 (#2923) * Bump histogram from 0.8.4 to 0.9.0 (#2935) * Bump jinja2 from 3.1.2 to 3.1.3 (#2922) * Bump memmap2 from 0.9.0 to 0.9.2 (#2882) * Bump memmap2 from 0.9.2 to 0.9.3 (#2889) * Bump memmap2 from 0.9.3 to 0.9.4 (#2958) * Bump mymindstorm/setup-emsdk from 13 to 14 (#2934) * Bump ouroboros from 0.18.1 to 0.18.2 (#2894) * Bump ouroboros from 0.18.2 to 0.18.3 (#2936) * Bump pypa/cibuildwheel from 2.16.2 to 2.16.4 (#2960) * Bump rayon from 1.8.0 to 1.8.1 (#2937) * Bump rkyv from 0.7.42 to 0.7.43 (#2880) * Bump serde from 1.0.194 to 1.0.195 (#2901) * Bump serde from 1.0.195 to 1.0.196 (#2956) * Bump serde_json from 1.0.108 to 1.0.110 (#2896) * Bump serde_json from 1.0.110 to 1.0.111 (#2902) * Bump serde_json from 1.0.111 to 1.0.113 (#2955) * Bump shlex from 1.1.0 to 1.3.0 (#2940) * Bump supercharge/redis-github-action from 1.7.0 to 1.8.0 (#2885) * Bump tempfile from 3.8.1 to 3.9.0 (#2893) * Bump thiserror from 1.0.50 to 1.0.51 (#2881) * Bump thiserror from 1.0.51 to 1.0.56 (#2897) * Bump wasm-bindgen from 0.2.89 to 0.2.90 (#2925) * Bump wasm-bindgen-test from 0.3.39 to 0.3.40 (#2938) * Bump web-sys from 0.3.66 to 0.3.67 (#2939) * Update pytest requirement from <7.5.0,>=6.2.4 to >=6.2.4,<8.1.0 (#2959)

bluegenes added 8 commits January 16, 2024 18:13

downsample within select

641d119

upd

be26f02

keep option

9918b5e

reorg scaled check

f0bb069

comment scaled logic

49ad9f5

rustfmt

450c767

simplify

2982d94

use retain properly; ds within storage

37164db

bluegenes added 7 commits January 17, 2024 12:32

rustfmt

e820e53

move ds logic back to Signature Select

869c7c4

Merge branch 'latest' into select-downsample

7401b95

actually replace sketch with ds version; add test

1e626f2

add cant downsample test

6779f0a

leave storage as-is

69b4b54

consolidate tests with other signature tests

986fc3d

bluegenes changed the title ~~WIP: in core, enable downsample within select~~ MRG: in core, enable downsample within select Jan 22, 2024

ctb approved these changes Jan 22, 2024

View reviewed changes

bluegenes and others added 4 commits January 22, 2024 15:53

Merge branch 'latest' into select-downsample

b780164

roll back byteorder for sourmash_plugin_branchwater (zip) compatibility

454a614

Merge branch 'select-downsample' of github.com:sourmash-bio/sourmash …

3ac5b22

…into select-downsample

also roll back once_cell and wasm_bindgen

efd1ee4

bluegenes force-pushed the select-downsample branch from d6bc50f to efd1ee4 Compare January 23, 2024 02:04

luizirber approved these changes Jan 23, 2024

View reviewed changes

src/core/src/signature.rs Outdated Show resolved Hide resolved

luizirber mentioned this pull request Jan 23, 2024

Add downsampling to select_sketch in rust #1292

Closed

bluegenes and others added 2 commits January 23, 2024 07:22

dbg print instead of eprintln

b51707e

Co-authored-by: Luiz Irber <luizirber@users.noreply.github.com>

Merge branch 'latest' into select-downsample

1b74d45

bluegenes merged commit 94b88cc into latest Jan 23, 2024
38 checks passed

bluegenes deleted the select-downsample branch January 23, 2024 19:24

bluegenes mentioned this pull request Jan 24, 2024

MRG: use updated rocksdb-integrated sourmash core sourmash-bio/sourmash_plugin_branchwater#134

Merged

ctb mentioned this pull request Feb 10, 2024

MRG: bump rust core version to r0.12.1 #2988

Merged

ctb mentioned this pull request Feb 10, 2024

MRG: 4.8.6 release branch #2963

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MRG: in core, enable downsample within select #2931

MRG: in core, enable downsample within select #2931

bluegenes commented Jan 17, 2024 •

edited

Loading

codecov bot commented Jan 17, 2024 •

edited

Loading

bluegenes commented Jan 22, 2024

ctb left a comment

luizirber left a comment

bluegenes commented Jan 23, 2024

MRG: in core, enable downsample within select #2931

MRG: in core, enable downsample within select #2931

Conversation

bluegenes commented Jan 17, 2024 • edited Loading

codecov bot commented Jan 17, 2024 • edited Loading

Codecov Report

bluegenes commented Jan 22, 2024

ctb left a comment

Choose a reason for hiding this comment

luizirber left a comment

Choose a reason for hiding this comment

bluegenes commented Jan 23, 2024

bluegenes commented Jan 17, 2024 •

edited

Loading

codecov bot commented Jan 17, 2024 •

edited

Loading