[INFRA] MATLAB test automation #43

Remi-Gau · 2022-04-07T11:24:38Z

Because this has not been merged, the results of the Github continuous integration woklfows are for now only viewable in the action tab of my fork

The initial commit of this PR is common to #44 (some changes are common to both PR and this might help prevent merge conflicts).

relates to #33

TODO

get code coverage
set up code coverage upload on codecov
Find an OS independent way to call MATLAB from the Makefile.

DONE

developers documentation
ensure that demo workflows shows as failed if any of the demo fails
add badges in the README
add Makefile (to help install test data, code cleaning...)
set up basic system tests
automate system tests via GITHUB CI
check if possible / useful to run tests with different OS / MATLAB versions
only run test in CI on push on master and PR on all branches

Common base for the matlab and python testing branch to minimize merge conflicts down the line

tests/test_glmsingleunit.m

.github/workflows/run_tests_matlab.yml

Remi-Gau · 2022-04-07T12:58:16Z

Matlab demos run in 45 minutes in CI (see here).

So running it on every commit of every pull-request might be a bit much.

Maybe a better approach could be to run it:

only on push to the master branch (so only once when the PR is merged)
at regular intervals with a CRON job (see here)

.github/workflows/run_tests_matlab.yml

kendrickkay · 2022-04-07T13:05:55Z

Matlab demos run in 45 minutes in CI (see here).

I see. The example scripts in the repository were intended to be illustrative/teaching and weren't meant to be super fast. In the one-off code thing I gave you I heavily subsetted the voxels (i think to 20 x 20 x 1), with the intention of running really quickly. When I ran it on my laptop, it took like 2-3 min i think. So, perhaps for these CI things, we should just use the tiny data case instead of the longer examples?

Remi-Gau · 2022-04-07T13:12:34Z

Matlab demos run in 45 minutes in CI (see here).

I see. The example scripts in the repository were intended to be illustrative/teaching and weren't meant to be super fast. In the one-off code thing I gave you I heavily subsetted the voxels (i think to 20 x 20 x 1), with the intention of running really quickly. When I ran it on my laptop, it took like 2-3 min i think. So, perhaps for these CI things, we should just use the tiny data case instead of the longer examples?

Yup the CI workflow that runs the tiny examples in CI takes less than 2 minutes: see here

But you can still set up another workflow to also run the demos automatically (say once a month) to make sure that a PR merged "recently" did not break anything.

kendrickkay · 2022-04-07T13:30:10Z

But you can still set up another workflow to also run the demos automatically (say once a month) to make sure that a PR merged "recently" did not break anything.

Aha, the 'cron' checking/testing is a cool idea, in the sense that it will periodically check and check and alert us if something doesn't seem right.

tests/test_GLMestimatesingletrial.m

Remi-Gau · 2022-04-07T16:09:29Z

.github/workflows/run_demos_matlab.yml

+# Uses the cron schedule for github actions
+#
+#   will run at 00H00 run on the 1 and 15 of every month
+#
+# https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#scheduled-events
+#
+#         ┌───────────── minute (0 - 59)
+#         │ ┌───────────── hour (0 - 23)
+#         │ │ ┌───────────── day of the month (1 - 31)
+#         │ │ │    ┌───────────── month (1 - 12 or JAN-DEC): * means all
+#         │ │ │    │ ┌───────────── day of the week (0 - 6 or SUN-SAT): * means all
+#         │ │ │    │ │
+#         │ │ │    │ │
+#         │ │ │    │ │
+# 
+# - cron "0 0 1,15 * *"
+
+on:
+  schedule:
+    - cron: "0 0 1,15 * *"


Demos will be run automatically every 15 days

run test demi CI set up through yml undo refactoring fix function name make sure demos run locally use same approach to run tests and demos in CI use similar calls for both tests and demos in CI

Remi-Gau · 2022-04-08T16:54:49Z

-[ ] ensure that demo workflows shows as failed if any of the demo fails

Currently not the workflows shows "green" ✔️ even when forcing the demos to fail.
https://github.com/Remi-Gau/GLMsingle/runs/5888747006?check_suite_focus=true

EDIT: now fixed by Remi-Gau@10161f0

Remi-Gau · 2022-04-08T19:29:30Z

CONTRIBUTING.md

+You can use the [`pre-commit` python package](https://pre-commit.com/) in this
+repo to make sure you only commit properly formatted files (for example `.yml`
+files).
+
+1. Install `pre-commit`
+
+```bash
+$ pip3 install pre-commit
+```
+
+It is also included in `requirements_dev.txt`, so it will installed by running:
+
+```bash
+$ pip3 install -r requirements_dev.txt
+```
+
+The `.pre-commit-config.yml` file defines the checks to run when committing
+files.
+
+1. Run the following command to install the `pre-commit` "hooks"
+
+```bash
+$ pre-commit install
+```


.yml files have a pretty strict syntax so using pre-commit can of great help to make sure you do not commit things YML files that can't be parsed.

Remi-Gau · 2022-04-08T19:35:18Z

CONTRIBUTING.md

+The [`miss_hit` python package](https://misshit.org/) is used to help ensure a
+consistent coding style for some of the MATLAB code.
+
+`miss_hit` can check code style, do a certain amount of automatic code
+reformating and prevent the code complexity from getting out of hand by running
+static code analysis (Static analysis can is a way to measure and track software
+quality metrics without additional code like tests).


I am a big fan of miss_hit to ensure a consistent MATLAB coding style when there are several contributors to the same repo.

The way things are set up now, it should not touch any of of the code in the matlab folder but should only act on the bit of code in tests and .github/workflows.

I don't want to touch to any code of GLMsingle with it because this is not in the scope of this PR.

But in any case, let me know how you feel about having any of miss_hit stuff in this repo, even for the test code base. I can remove it if you want.

Remi-Gau · 2022-04-08T19:36:52Z

README.md

+[![MATLAB test](https://github.com/cvnlab/GLMsingle/actions/workflows/run_tests_matlab.yml/badge.svg)](https://github.com/cvnlab/GLMsingle/actions/workflows/run_tests_matlab.yml)
+[![MATLAB demos](https://github.com/cvnlab/GLMsingle/actions/workflows/run_demos_matlab.yml/badge.svg)](https://github.com/cvnlab/GLMsingle/actions/workflows/run_demos_matlab.yml)
+


FYI those badges should be valid after merging even it the URL are currently broken.

Remi-Gau · 2022-04-08T19:38:14Z

README.md

-## Example scripts
-
-We provide a number of example scripts that demonstrate usage of GLMsingle. You can browse these example scripts here:
-
-(Python Example 1 - event-related design) https://htmlpreview.github.io/?https://github.com/kendrickkay/GLMsingle/blob/main/examples/example1.html
-
-(Python Example 2 - block design) https://htmlpreview.github.io/?https://github.com/kendrickkay/GLMsingle/blob/main/examples/example2.html
-
-(MATLAB Example 1 - event-related design) https://htmlpreview.github.io/?https://github.com/kendrickkay/GLMsingle/blob/main/matlab/examples/example1preview/example1.html
-
-(MATLAB Example 2 - block design) https://htmlpreview.github.io/?https://github.com/kendrickkay/GLMsingle/blob/main/matlab/examples/example2preview/example2.html
-
-If you would like to run these example scripts, the Python versions are available in `/GLMsingle/examples`, and the MATLAB versions are available in `/GLMsingle/matlab/examples`. Each notebook contains a full walkthrough of the process of loading an example dataset and design matrix, estimating neural responses using GLMsingle, estimating the reliability of responses at each voxel, and comparing those achieved via GLMsingle to those achieved using a baseline GLM.
-


FYI Reordered the README to have all the install instructions first for MATLAB and Python and then the examples. Can change it back.

Remi-Gau · 2022-04-08T20:02:18Z

.github/workflows/run_tests_matlab.yaml

+    strategy:
+      matrix:
+        os: [ubuntu-latest] # "macos-latest" or "windows-latest" don't work (yet?)
+        matlab-version: ["R2020a"] # add more versions if needed: "R2021a"
+      fail-fast: false  # Don't cancel all jobs if one fails


If we ever wanted to test on different OS or matlab version this could simply be changed in there.

Currently Matlab github actions only support linux.

Remi-Gau · 2022-04-08T20:06:30Z

@kendrickkay this is 95% finished for me. So you can have a look at the whole thing and ask for further clarifications / documentation, changes.

The unticked boxes in the TODO list at the top are things that I would like to have but may require a bit more work and could be addressed in separate PRs as they are not essential for the core purpose of this PR.

Remi-Gau added 4 commits April 7, 2022 13:12

Add common tools for MATLAB and Python testing

af934b4

Common base for the matlab and python testing branch to minimize merge conflicts down the line

add smoke tests matlab

20e6fd9

add matlab test automation

918091b

add matlab test demo automation

aee01dc

Remi-Gau commented Apr 7, 2022

View reviewed changes

tests/test_glmsingleunit.m Outdated Show resolved Hide resolved

Remi-Gau commented Apr 7, 2022

View reviewed changes

.github/workflows/run_tests_matlab.yml Outdated Show resolved Hide resolved

Remi-Gau commented Apr 7, 2022

View reviewed changes

.github/workflows/run_tests_matlab.yml Outdated Show resolved Hide resolved

Remi-Gau added 5 commits April 7, 2022 16:26

add expected data and an assert to system test

faba452

rename test file

b886a8b

add documentation

04508a5

lint

b9df9f2

remove unneeded upload coverage artefact from CI workflow

0eff103

Remi-Gau commented Apr 7, 2022

View reviewed changes

tests/test_GLMestimatesingletrial.m Show resolved Hide resolved

Remi-Gau added 3 commits April 7, 2022 17:44

run tests matlab demos with CRON job on 1rt and 15th of every month

3ab086b

add assert on value of R2

ef3920e

add doc in CI yml and scripts

8284581

Remi-Gau commented Apr 7, 2022

View reviewed changes

Remi-Gau added 4 commits April 7, 2022 18:13

fix typo

61cfe71

add matlab tests badge to README

bd5d8fa

add matlab demo badge to README

956da88

run demos with MOxUnit in CI so that demo2 is run even if demo1 fails

9def9a2

run test demi CI set up through yml undo refactoring fix function name make sure demos run locally use same approach to run tests and demos in CI use similar calls for both tests and demos in CI

Remi-Gau force-pushed the matlab_test_automation branch from 5c1cdd0 to 9def9a2 Compare April 8, 2022 16:45

force demos to fail to make sure this is picked by CI

c4530c3

Remi-Gau added 2 commits April 8, 2022 18:56

add a check log to try to show workflow as failed

10161f0

remove forced errors in demo and reset to run demos bimonthly

f569dd6

Remi-Gau added 8 commits April 8, 2022 19:15

add doc on running tests and demos in CI

26389af

miss_hit lint CI code

56807a6

add miss hit and linting documentation

07da5c3

add pre-commit doc

1908552

update contributors doc

70d205e

add doc on generating and chekcing test data

731ac95

update repo map

a312579

switch to using png image in doc

b35bec6

Remi-Gau commented Apr 8, 2022

View reviewed changes

Remi-Gau added 2 commits April 8, 2022 21:52

try testing different os and matlab version

ca40488

only run tests on ubuntu latest with matlab 2020a

882cf8e

Remi-Gau commented Apr 8, 2022

View reviewed changes

Remi-Gau changed the title ~~[WIP][INFRA] MATLAB test automation~~ [INFRA] MATLAB test automation Apr 8, 2022

Remi-Gau marked this pull request as ready for review April 8, 2022 20:02

Remi-Gau requested a review from kendrickkay April 8, 2022 20:02

This was referenced Apr 9, 2022

[DOC] Add sphinx documentation and serve on read the docs #45

Merged

[WIP][ENH] add Dockerfile for python version og GLM single #46

Draft

Update .github/workflows/run_tests_matlab.yaml

4ae81fd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INFRA] MATLAB test automation #43

[INFRA] MATLAB test automation #43

Remi-Gau commented Apr 7, 2022 •

edited

Loading

Remi-Gau commented Apr 7, 2022 •

edited

Loading

kendrickkay commented Apr 7, 2022

Remi-Gau commented Apr 7, 2022

kendrickkay commented Apr 7, 2022

Remi-Gau Apr 7, 2022

Remi-Gau commented Apr 8, 2022 •

edited

Loading

Remi-Gau Apr 8, 2022

Remi-Gau Apr 8, 2022 •

edited

Loading

Remi-Gau Apr 8, 2022

Remi-Gau Apr 8, 2022

Remi-Gau Apr 8, 2022

Remi-Gau commented Apr 8, 2022

		[![MATLAB test](https://github.com/cvnlab/GLMsingle/actions/workflows/run_tests_matlab.yml/badge.svg)](https://github.com/cvnlab/GLMsingle/actions/workflows/run_tests_matlab.yml)
		[![MATLAB demos](https://github.com/cvnlab/GLMsingle/actions/workflows/run_demos_matlab.yml/badge.svg)](https://github.com/cvnlab/GLMsingle/actions/workflows/run_demos_matlab.yml)

[INFRA] MATLAB test automation #43

Are you sure you want to change the base?

[INFRA] MATLAB test automation #43

Conversation

Remi-Gau commented Apr 7, 2022 • edited Loading

TODO

DONE

Remi-Gau commented Apr 7, 2022 • edited Loading

kendrickkay commented Apr 7, 2022

Remi-Gau commented Apr 7, 2022

kendrickkay commented Apr 7, 2022

Remi-Gau Apr 7, 2022

Choose a reason for hiding this comment

Remi-Gau commented Apr 8, 2022 • edited Loading

Remi-Gau Apr 8, 2022

Choose a reason for hiding this comment

Remi-Gau Apr 8, 2022 • edited Loading

Choose a reason for hiding this comment

Remi-Gau Apr 8, 2022

Choose a reason for hiding this comment

Remi-Gau Apr 8, 2022

Choose a reason for hiding this comment

Remi-Gau Apr 8, 2022

Choose a reason for hiding this comment

Remi-Gau commented Apr 8, 2022

Remi-Gau commented Apr 7, 2022 •

edited

Loading

Remi-Gau commented Apr 7, 2022 •

edited

Loading

Remi-Gau commented Apr 8, 2022 •

edited

Loading

Remi-Gau Apr 8, 2022 •

edited

Loading