Update ctrDetect Binaries #2666

kousu · 2020-04-21T05:14:16Z

Manually version bump ctrDetect to match upstream's distribution.

This should be a minor change. It is in preparation for better automating the build which will download from M. Sdika's upstream directly.

How The Packages Were Built

$ curl -JLO https://osf.io/mka78/?action=download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   459  100   459    0     0   1577      0 --:--:-- --:--:-- --:--:--  1577
100 74.6M  100 74.6M    0     0  2870k      0  0:00:26  0:00:26 --:--:-- 3842k
curl: Saved to filename '20200420_sct_binaries_linux.tar.gz'
$ curl -JLO https://osf.io/dn67h/?action=download 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   459  100   459    0     0   1645      0 --:--:-- --:--:-- --:--:--  1645
100 57.8M  100 57.8M    0     0  3473k      0  0:00:17  0:00:17 --:--:-- 3851k
curl: Saved to filename '20200420_sct_binaries_osx.tar.gz'

Follow https://github.com/neuropoly/spinalcordtoolbox/wiki/Compile-binaries -> https://www.creatis.insa-lyon.fr/site7/en/ctrDetect ->

$ curl -JLO http://www.creatis.insa-lyon.fr/~sdika/soft/ctrDetect-v1_x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   279  100   279    0     0    690      0 --:--:-- --:--:-- --:--:--   692
100 33.9M  100 33.9M    0     0  1092k      0  0:00:31  0:00:31 --:--:-- 1131k
$ curl -JLO http://www.creatis.insa-lyon.fr/~sdika/soft/ctrDetect-v1_macos10.11.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   283  100   283    0     0    686      0 --:--:-- --:--:-- --:--:--   685
100  9.8M  100  9.8M    0     0  1966k      0  0:00:05  0:00:05 --:--:-- 2520k

Unpack the old version, update ctrDetect, and repack:

$ mkdir binaries_osx; tar -zxvf 20200420_sct_binaries_osx.tar.gz -C binaries_osx
./
./isct_dice_coefficient
./isct_antsRegistration
./isct_spine_detect
./isct_ComposeMultiTransform
./isct_train_svm
./isct_propseg
./isct_antsSliceRegularizedRegistration
./isct_antsApplyTransforms
$ tar -zxvf ctrDetect-v1_macos10.11.tar.gz 
ctrDetect/
ctrDetect/LICENSE.txt
ctrDetect/LICENSE_opencv.txt
ctrDetect/README.txt
ctrDetect/README.txt~
ctrDetect/classif_CC.yml
ctrDetect/rectifyCtr.py
ctrDetect/spine_detect
ctrDetect/spine_detect.bash
ctrDetect/spine_train_svm
ctrDetect/spine_train_svm.bash
$ cp -p ctrDetect/spine_detect binaries_osx/isct_spine_detect  # -p to preserve the timestamp
$ cp -p ctrDetect/spine_train_svm binaries_osx/isct_train_svm
$ ls -l binaries_osx/
total 197284
-rwxr-xr-x 1 kousu kousu 32203680 Apr 19 14:44 isct_antsApplyTransforms
-rwxr-xr-x 1 kousu kousu 48876288 Apr 19 14:44 isct_antsRegistration
-rwxr-xr-x 1 kousu kousu 24320568 Apr 19 14:44 isct_antsSliceRegularizedRegistration
-rwxr-xr-x 1 kousu kousu 22713500 Apr 19 14:44 isct_ComposeMultiTransform
-rwxr-xr-x 1 kousu kousu  9589820 Mar  2  2016 isct_dice_coefficient
-rwxr-xr-x 1 kousu kousu 36514576 Sep 18  2016 isct_propseg
-rwxr-xr-x 1 kousu kousu 13873744 Apr 11  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 13909424 Apr 11  2017 isct_train_svm
$ tar -czvf 20200421_sct_binaries_osx.tar.gz -C binaries_osx/ .
./
./isct_antsRegistration
./isct_propseg
./isct_train_svm
./isct_dice_coefficient
./isct_ComposeMultiTransform
./isct_antsApplyTransforms
./isct_spine_detect
./isct_antsSliceRegularizedRegistration

Now do the Linux version:

$ mkdir binaries_linux; tar -zxvf 20200420_sct_binaries_linux.tar.gz -C binaries_linux
./
./isct_dice_coefficient
./isct_antsRegistration
./isct_spine_detect
./isct_ComposeMultiTransform
./isct_train_svm
./isct_propseg
./isct_antsSliceRegularizedRegistration
./isct_antsApplyTransforms   
$ tar -zxvf ctrDetect-v1_x86_64.tar.gz 
ctrDetect/
ctrDetect/libc.so.6
ctrDetect/classif_CC.yml
ctrDetect/README.txt~
ctrDetect/LICENSE_opencv.txt
ctrDetect/spine_detect.bash
ctrDetect/spine_detect
ctrDetect/spine_train_svm.bash
ctrDetect/libc.so
ctrDetect/README.txt
ctrDetect/spine_train_svm
ctrDetect/LICENSE.txt
$ ls -l binaries_linux/
total 246112
-rwxr-xr-x 1 kousu kousu 28630112 Apr 19 15:08 isct_antsApplyTransforms
-rwxr-xr-x 1 kousu kousu 44594624 Apr 19 15:08 isct_antsRegistration
-rwxr-xr-x 1 kousu kousu 21435952 Apr 19 15:08 isct_antsSliceRegularizedRegistration
-rwxr-xr-x 1 kousu kousu 19880216 Apr 19 15:08 isct_ComposeMultiTransform
-rwxr-xr-x 1 kousu kousu 10439872 Mar  2  2016 isct_dice_coefficient
-rwxr-xr-x 1 kousu kousu 20504754 Jul  5  2016 isct_propseg
-rwxr-xr-x 1 kousu kousu 53186384 Apr  9  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 53331105 Dec  4  2018 isct_train_svm
$ cp -p ctrDetect/spine_detect binaries_linux/isct_spine_detect 
$ cp -p ctrDetect/spine_train_svm binaries_linux/isct_train_svm 
$ ls -l binaries_linux/
total 246100
-rwxr-xr-x 1 kousu kousu 28630112 Apr 19 15:08 isct_antsApplyTransforms
-rwxr-xr-x 1 kousu kousu 44594624 Apr 19 15:08 isct_antsRegistration
-rwxr-xr-x 1 kousu kousu 21435952 Apr 19 15:08 isct_antsSliceRegularizedRegistration
-rwxr-xr-x 1 kousu kousu 19880216 Apr 19 15:08 isct_ComposeMultiTransform
-rwxr-xr-x 1 kousu kousu 10439872 Mar  2  2016 isct_dice_coefficient
-rwxr-xr-x 1 kousu kousu 20504754 Jul  5  2016 isct_propseg
-rwxr-xr-x 1 kousu kousu 53172736 Apr 19  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 53331105 Apr 19  2017 isct_train_svm
$ tar -czvf 20200421_sct_binaries_linux.tar.gz -C binaries_linux/ .
./
./isct_antsRegistration
./isct_propseg
./isct_train_svm
./isct_dice_coefficient
./isct_ComposeMultiTransform
./isct_antsApplyTransforms
./isct_spine_detect
./isct_antsSliceRegularizedRegistration

Checksums:

$ sha256sum 20200421_sct_binaries_*
e1968ddf15094f860777cbe2167870e2100662db951da78fcb79f8bf6190c394  20200421_sct_binaries_linux.tar.gz
b3365d0e125a7f9f5bd1726acfb75641b3e1c33e50acfc9ac4bb274cbaea5fc2  20200421_sct_binaries_osx.tar.gz

kousu · 2020-04-21T05:30:46Z

Ah, that's weird. This actually downgraded ctrDetect, if I can trust the timestamps. The OS X version we were distributing was built a day after the OS X 10.11 version, and the Linux version we were distributing includes one file 10 days older but another file a year-and-a-half newer than M. Sdika's published version.

Comparing timestamps

Re-extract the ctrDetect binaries, old and new and for both platforms:

$ mkdir 20200420_sct_binaries_linux; tar -C 20200420_sct_binaries_linux -zxvf 20200420_sct_binaries_linux.tar.gz ./isct_train_svm ./isct_spine_detects
./isct_spine_detect
./isct_train_svm
$ mkdir 20200421_sct_binaries_linux; tar -C 20200421_sct_binaries_linux -zxvf 20200421_sct_binaries_linux.tar.gz ./isct_train_svm ./isct_spine_detect
./isct_train_svm
./isct_spine_detect
$ mkdir 20200420_sct_binaries_osx; tar -C 20200420_sct_binaries_osx -zxvf 20200420_sct_binaries_osx.tar.gz ./isct_train_svm ./isct_spine_detect
./isct_spine_detect
./isct_train_svm
$ mkdir 20200421_sct_binaries_osx; tar -C 20200421_sct_binaries_osx -zxvf 20200421_sct_binaries_osx.tar.gz ./isct_train_svm ./isct_spine_detect
./isct_train_svm
./isct_spine_detect

Compare dates:

$ ls -l 2020042*_sct_binaries_linux
20200420_sct_binaries_linux:
total 104028
-rwxr-xr-x 1 kousu kousu 53186384 Apr  9  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 53331105 Dec  4  2018 isct_train_svm

20200421_sct_binaries_linux:
total 104012
-rwxr-xr-x 1 kousu kousu 53172736 Apr 19  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 53331105 Apr 19  2017 isct_train_svm
$ ls -l 2020042*_sct_binaries_osx
20200420_sct_binaries_osx:
total 27504
-rwxr-xr-x 1 kousu kousu 14071264 Apr 12  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 14090052 Apr 12  2017 isct_train_svm

20200421_sct_binaries_osx:
total 27136
-rwxr-xr-x 1 kousu kousu 13873744 Apr 11  2017 isct_spine_detect
-rwxr-xr-x 1 kousu kousu 13909424 Apr 11  2017 isct_train_svm

I tried looking in the older(?) OS X version to see if that explains part of it, but it does not: what we have been distributing is neither.

Comparing OS X contents

Extract binaries:

20200420_sct_binaries_osx.tar.gz    20200421_sct_binaries_osx.tar.gz    ctrDetect-v1_x86_64.tar.gz
$ tar -zxvf ctrDetect-v1_macos10.11.tar.gz 
ctrDetect/
ctrDetect/LICENSE.txt
ctrDetect/LICENSE_opencv.txt
ctrDetect/README.txt
ctrDetect/README.txt~
ctrDetect/classif_CC.yml
ctrDetect/rectifyCtr.py
ctrDetect/spine_detect
ctrDetect/spine_detect.bash
ctrDetect/spine_train_svm
ctrDetect/spine_train_svm.bash
$ tar -zxvf ctrDetect-v1_macos10.7.tar.gz 
ctrDetect10.7/
ctrDetect10.7/LICENSE.txt
ctrDetect10.7/LICENSE_opencv.txt
ctrDetect10.7/README.txt
ctrDetect10.7/classif_CC.yml
ctrDetect10.7/rectifyCtr.py
ctrDetect10.7/spine_detect
ctrDetect10.7/spine_detect.bash
ctrDetect10.7/spine_train_svm
ctrDetect10.7/spine_train_svm.bash

Compare hashes:

$ sha256sum 20200420_sct_binaries_osx/*train_svm ctrDetect*/*train_svm
f1bc1ceb4db2ec2ab046800648d5dcc7eb547736bd3559e6e9c17ed44f1ed1f8  20200420_sct_binaries_osx/isct_train_svm
58df50b57d67243af92f10fbfd079ff35f32081fd150866a36424e46592c18cd  ctrDetect10.7/spine_train_svm
1ade423edf0ef085edabea1ba33b900b92f22309bef78443ed06293cb07dce1e  ctrDetect/spine_train_svm
$ sha256sum 20200420_sct_binaries_osx/*spine_detect ctrDetect*/*spine_detect
07d3f91562878ad9b3375385ebcf093d0ae72d399427dbc7dabd5795c0e4374c  20200420_sct_binaries_osx/isct_spine_detect
d2c11bde2a0c4bb77b1eddcc52917167589ef78ffd216c81684c3798fa6db379  ctrDetect10.7/spine_detect
004b430bd3646d591aa236699f446e42bc974974db36f580d73c1795e9af9d13  ctrDetect/spine_detect

Do we have an idea where these binaries came from?

This is temporary; for comparing results against #2666.

This is temporary; for comparing results against #2666. Update .travis.yml

This is temporary; for comparing results against #2666.

kousu · 2020-05-05T02:20:15Z

Despite this downgrading the binaries, I want to merge it for consistency's sake -- particularly, so that the build scripts for #2668 can refer to upstream directly instead of us having to keep a weird pseudo-mirror that no one else has just of those.

It turned out that all the inconsistencies above were just problems in #2642, that are fixed now, so whatever downgrade this is it doesn't change the numbers appreciably.

codecov · 2022-05-06T13:53:32Z

Codecov Report

Merging #2666 (c49deeb) into master (528dfe2) will decrease coverage by 0.00%.
The diff coverage is n/a.

Flag	Coverage Δ
api-tests	`22.63% <ø> (-0.01%)`	⬇️
cli-tests	`58.60% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
spinalcordtoolbox/scripts/sct_download_data.py	`83.87% <ø> (ø)`
spinalcordtoolbox/utils/sys.py	`63.52% <0.00%> (-0.30%)`	⬇️

joshuacwnewton · 2022-05-06T14:24:58Z

I've cleaned up this PR a bit so that it is easier to understand, and I want to summarize the changes I made.

This PR originally had 8 commits. However, 7/8 of those commits were actually just duplicates of PR #2642, which had already been merged. The one remaining (unique) commit was: 9d09785.
This PR originally had a lot of discussion about numerical stability. However, as @kousu mentioned in this comment, the discussion was only relevant to the 7 duplicate commits that were already merged, and not to the 1 commit that remains.

So, I discarded the 7 duplicate commits, then rebased the changes from that 1 commit onto master.

joshuacwnewton · 2022-05-06T21:08:10Z

Closing this PR as it conflicts with #2668, and I believe that it's more likely that it will be superseded by #3788/#2668 than to be merged standalone.

kousu marked this pull request as draft April 21, 2020 05:14

This was referenced Apr 21, 2020

License binaries #2667

Closed

Use binaries that are packaged by the spinalcordtoolbox-binaries repo #2668

Merged

Packaging Dependencies #2669

Open

kousu force-pushed the ko/new-ctrDetect-binaries branch from 736b24b to 5e5d162 Compare April 26, 2020 20:03

This comment was marked as outdated.

Sign in to view

kousu added a commit that referenced this pull request Apr 27, 2020

testing

6012d48

This is temporary; for comparing results against #2666.

This comment was marked as outdated.

Sign in to view

kousu added a commit that referenced this pull request Apr 27, 2020

testing

1104110

This is temporary; for comparing results against #2666. Update .travis.yml

kousu added a commit that referenced this pull request May 4, 2020

testing

058f8e0

This is temporary; for comparing results against #2666.

This comment was marked as outdated.

Sign in to view

kousu force-pushed the ko/new-ctrDetect-binaries branch from 6cbe083 to 9d09785 Compare May 5, 2020 01:52

This was referenced May 5, 2020

Updated ANTs binaries to solve slow processing in some systems #2642

Merged

sct_download_data safety checks #2689

Merged

This comment was marked as resolved.

Sign in to view

This was referenced Apr 13, 2022

Priorities for SCT v5.7 Release #3765

Closed

Resolve SCT's unfinished draft pull requests #3768

Closed

joshuacwnewton self-assigned this May 5, 2022

Update ctrDetect binaries.

c49deeb

joshuacwnewton force-pushed the ko/new-ctrDetect-binaries branch from 9d09785 to c49deeb Compare May 6, 2022 13:42

joshuacwnewton mentioned this pull request May 6, 2022

Use GitHub Actions workflows to automate building/packaging SCT's binary dependencies #3788

Closed

11 tasks

joshuacwnewton closed this May 6, 2022

joshuacwnewton deleted the ko/new-ctrDetect-binaries branch June 16, 2022 15:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ctrDetect Binaries #2666

Update ctrDetect Binaries #2666

kousu commented Apr 21, 2020 •

edited by joshuacwnewton

Loading

kousu commented Apr 21, 2020 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

kousu commented May 5, 2020

This comment was marked as resolved.

This comment was marked as resolved.

codecov bot commented May 6, 2022

joshuacwnewton commented May 6, 2022

joshuacwnewton commented May 6, 2022 •

edited

Loading

Update ctrDetect Binaries #2666

Update ctrDetect Binaries #2666

Conversation

kousu commented Apr 21, 2020 • edited by joshuacwnewton Loading

kousu commented Apr 21, 2020 • edited Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

kousu commented May 5, 2020

This comment was marked as resolved.

This comment was marked as resolved.

codecov bot commented May 6, 2022

Codecov Report

joshuacwnewton commented May 6, 2022

joshuacwnewton commented May 6, 2022 • edited Loading

kousu commented Apr 21, 2020 •

edited by joshuacwnewton

Loading

kousu commented Apr 21, 2020 •

edited

Loading

joshuacwnewton commented May 6, 2022 •

edited

Loading