From f21e9b2b2aea89878ecc3e0a77f3a19d49a3c6ef Mon Sep 17 00:00:00 2001 From: Remi Gau Date: Tue, 31 Aug 2021 17:23:20 +0200 Subject: [PATCH] use macro to render examples in a "tree" like fashion (#837) * Initial work. * update code to render simple examples (list and simple dictionaries) * create a separate module for example creation * start applying to spec * remove dead code * apply to MRI page * fix tabulation on scans.tsv template * apply example macro to longitudinal page * apply to common principles * revert changes on scans.tsv templates addressed in #805 * remove extra line after listing directory content * only use python dictionaries for examples rendering * update examples common principles * update example longitudinal page * update examples MRI page * update examples MEG * update examples iEEG * update examples task * update physio example * update examples common data types derivatives * update examples imaging datatypes * change "participant-label" to "label" in derivatives templates * update examples meg file formats * update qmri examples * reduce width prefixes * add a flag to be able to not use "pipes" prefix * switch use_pipe flag to off when building pdf * create an examplecode module * add demo jupyter notebook for tree example * add doc in code * add doc to CONTRIBUTING * remove non macro examples * fix typos (comment padding and missing folders) * temporary commit * remove comment * fix import * fix markdown linting errors * Apply suggestions from code review Co-authored-by: Stefan Appelhoff * Apply suggestions from code review Co-authored-by: Stefan Appelhoff * remove jupyter notebook output * revert changes to import in schema code init * revert change of import of main in macro module this style of import is required for the building of the pdf Co-authored-by: Taylor Salo Co-authored-by: Stefan Appelhoff --- CONTRIBUTING.md | 38 ++- mkdocs.yml | 2 +- pdf_build_src/process_markdowns.py | 8 +- src/02-common-principles.md | 214 +++++++------ .../01-magnetic-resonance-imaging-data.md | 168 ++++++---- .../02-magnetoencephalography.md | 43 ++- .../04-intracranial-electroencephalography.md | 26 +- .../05-task-events.md | 50 +-- ...logical-and-other-continuous-recordings.md | 46 ++- src/05-derivatives/02-common-data-types.md | 93 ++++-- src/05-derivatives/03-imaging.md | 244 +++++++++------ src/06-longitudinal-and-multi-site-studies.md | 95 +++--- src/99-appendices/06-meg-file-formats.md | 290 +++++++++++------- src/99-appendices/11-qmri.md | 229 ++++++++------ tools/examplecode/__init__.py | 6 + tools/examplecode/example.py | 79 +++++ tools/filetree_example.ipynb | 163 ++++++++++ tools/mkdocs_macros_bids/__init__.py | 19 ++ .../macros.py | 28 +- .../main.py | 8 +- tools/mkdocs_macros_bidsschema/__init__.py | 3 - tools/schemacode/__init__.py | 3 +- 22 files changed, 1248 insertions(+), 607 deletions(-) create mode 100644 tools/examplecode/__init__.py create mode 100644 tools/examplecode/example.py create mode 100644 tools/filetree_example.ipynb create mode 100644 tools/mkdocs_macros_bids/__init__.py rename tools/{schemacode => mkdocs_macros_bids}/macros.py (81%) rename tools/{mkdocs_macros_bidsschema => mkdocs_macros_bids}/main.py (86%) delete mode 100644 tools/mkdocs_macros_bidsschema/__init__.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b3c51232e..0e30b4a746 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -228,12 +228,11 @@ to render parts of the BIDS specification from the BIDS schema. Macros make it easy to achieve a consistent style throughout the specification, and changing a given macro will automatically change all appropriate paragraphs in the specification. - For example, all tables on BIDS metadata are generated via macros that make use of data in the [yaml files](src/schema/metadata) in the [schema](src/schema/README.md). -The macros are written in Python -(see the folders [tools/schemacode](tools/schemacode) and [tools/mkdocs_macros_bidsschema](tools/mkdocs_macros_bidsschema)), +These macros are written in Python +(see the folders [tools/schemacode](tools/schemacode) and [tools/mkdocs_macros_bids](tools/mkdocs_macros_bids)), and are called directly in the Markdown document where you want the output of the macro to be inserted. For example: @@ -267,6 +266,39 @@ by specifying it in the macro call: ) }} ``` +### Writing folder content examples + +We also use macros to have a consistent style to render the examples of folder contents. + +These code for these macros are in the folder [tools/schemacode](tools/schemacode). + +To insert examples in the code you have make calls to the macro like this: + +``` +{{ MACROS___make_filetree_example( + + { + "sub-01": { + "func": { + "sub-control01_task-nback_bold.json": "", + }, + } + } + +) }} +``` + +And this will be turned into this. + +```Text +└─ sub-01/ + └─ func/ + └─ sub-control01_task-nback_bold.json +``` + +When you have complex files and folder structure, we suggest you use this +[Jupyter notebook](tools/filetree_example.ipynb) for sandboxing your example +before you insert the macro call into the markdown document. ## Building the specification using mkdocs diff --git a/mkdocs.yml b/mkdocs.yml index 34dbb28049..59c29c098a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,7 +19,7 @@ plugins: +extra_css: - css/watermark.css - macros: - module_name: tools/mkdocs_macros_bidsschema/main + module_name: tools/mkdocs_macros_bids/main docs_dir: 'src' use_directory_urls: false nav: diff --git a/pdf_build_src/process_markdowns.py b/pdf_build_src/process_markdowns.py index c39640c77a..53d9481d29 100644 --- a/pdf_build_src/process_markdowns.py +++ b/pdf_build_src/process_markdowns.py @@ -16,7 +16,7 @@ import numpy as np sys.path.append("../tools/") -from schemacode import macros # noqa (used in "eval" call later on) +from mkdocs_macros_bids import macros # noqa (used in "eval" call later on) def run_shell_cmd(command): @@ -453,6 +453,12 @@ def process_macros(duplicated_src_dir_path): "MACROS___", "macros." ) + # switch "use_pipe" flag OFF to render examples + if "make_filetree_example" in function_string: + function_string = function_string.replace( + ")", + ", False)" + ) # Run the function to get the output new = eval(function_string) # Replace the code snippet with the function output diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 9143720496..c8c5b306c6 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -254,21 +254,24 @@ However, in the case that these data are to be included: Alternatively one can organize their data in the following way -```Text -my_dataset/ - sourcedata/ - ... - rawdata/ - dataset_description.json - participants.tsv - sub-01/ - sub-02/ - ... - derivatives/ - pipeline_1/ - pipeline_2/ - ... -``` +{{ MACROS___make_filetree_example( + { + "my_dataset-1": { + "sourcedata": "", + "...": "", + "sourcedata": { + "sub-01": {}, + "sub-02": {}, + "...": "", + }, + "derivatives": { + "pipeline_1": {}, + "pipeline_2": {}, + "...": "", + }, + } + } +) }} In this example, where `sourcedata` and `derivatives` are not nested inside `rawdata`, **only the `rawdata` subfolder** needs to be a BIDS-compliant @@ -344,23 +347,25 @@ Derivatives can be stored/distributed in two ways: Example of a derivative dataset including the raw dataset as source: - ```Plain - my_processed_data/ - code/ - processing_pipeline-1.0.0.img - hpc_submitter.sh - ... - sourcedata/ - dataset_description.json - participants.tsv - sub-01/ - sub-02/ - ... - dataset_description.json - sub-01/ - sub-02/ - ... - ``` + {{ MACROS___make_filetree_example( + { + "my_processed_data": { + "code": { + "processing_pipeline-1.0.0.img": "", + "hpc_submitter.sh": "", + "...": "", + }, + "sourcedata": { + "sub-01": {}, + "sub-02": {}, + "...": "", + }, + "sub-01": {}, + "sub-02": {}, + "...": "", + } + } + ) }} Throughout this specification, if a section applies particularly to derivatives, then Case 1 will be assumed for clarity in templates and examples, but removing @@ -403,17 +408,23 @@ specific to a participant is to be declared only at top level of dataset for exa Example 1: Two JSON files that are erroneously at the same level -```Text -sub-01/ - ses-test/ - sub-01_ses-test_task-overtverbgeneration_bold.json - sub-01_ses-test_task-overtverbgeneration_run-2_bold.json - anat/ - sub-01_ses-test_T1w.nii.gz - func/ - sub-01_ses-test_task-overtverbgeneration_run-1_bold.nii.gz - sub-01_ses-test_task-overtverbgeneration_run-2_bold.nii.gz -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "ses-test":{ + "sub-01_ses-test_task-overtverbgeneration_bold.json": "", + "sub-01_ses-test_task-overtverbgeneration_run-2_bold.json": "", + "anat": { + "sub-01_ses-test_T1w.nii.gz": "", + }, + "func": { + "sub-01_ses-test_task-overtverbgeneration_run-1_bold.nii.gz": "", + "sub-01_ses-test_task-overtverbgeneration_run-2_bold.nii.gz": "", + } + } + } + } +) }} In the above example, two JSON files are listed under `sub-01/ses-test/`, which are each applicable to @@ -424,16 +435,20 @@ should have been under `sub-01/ses-test/func/`. Example 2: Multiple `run`s and `rec`s with same acquisition (`acq`) parameters -```Text -sub-01/ - anat/ - func/ - sub-01_task-xyz_acq-test1_run-1_bold.nii.gz - sub-01_task-xyz_acq-test1_run-2_bold.nii.gz - sub-01_task-xyz_acq-test1_rec-recon1_bold.nii.gz - sub-01_task-xyz_acq-test1_rec-recon2_bold.nii.gz - sub-01_task-xyz_acq-test1_bold.json -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "anat": {}, + "func": { + "sub-01_task-xyz_acq-test1_run-1_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_run-2_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_rec-recon1_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_rec-recon2_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_bold.json": "", + } + } + } +) }} For the above example, all NIfTI files are acquired with same scanning parameters (`acq-test1`). Hence a JSON file describing the acq parameters will @@ -443,16 +458,20 @@ will be applicable to all task runs with `test1` acquisition parameter. Example 3: Multiple JSON files at different levels for same task and acquisition parameters -```Text -task-xyz_acq-test1_bold.json -sub-01/ - anat/ - func/ - sub-01_task-xyz_acq-test1_run-1_bold.nii.gz - sub-01_task-xyz_acq-test1_rec-recon1_bold.nii.gz - sub-01_task-xyz_acq-test1_rec-recon2_bold.nii.gz - sub-01_task-xyz_acq-test1_bold.json -``` +{{ MACROS___make_filetree_example( + { + "task-xyz_acq-test1_bold.json": "", + "sub-01": { + "anat": {}, + "func": { + "sub-01_task-xyz_acq-test1_run-1_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_rec-recon1_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_rec-recon2_bold.nii.gz": "", + "sub-01_task-xyz_acq-test1_bold.json": "", + } + } + } +) }} In the above example, the fields from the `task-xyz_acq-test1_bold.json` file at the top directory will apply to all bold runs. However, if there is a key @@ -732,37 +751,44 @@ This is an example of the folder and file structure. Because there is only one session, the session level is not required by the format. For details on individual files see descriptions in the next section: -```Text -sub-control01/ - sub-control01_scans.tsv - anat/ - sub-control01_T1w.nii.gz - sub-control01_T1w.json - sub-control01_T2w.nii.gz - sub-control01_T2w.json - func/ - sub-control01_task-nback_bold.nii.gz - sub-control01_task-nback_bold.json - sub-control01_task-nback_events.tsv - sub-control01_task-nback_physio.tsv.gz - sub-control01_task-nback_physio.json - sub-control01_task-nback_sbref.nii.gz - dwi/ - sub-control01_dwi.nii.gz - sub-control01_dwi.bval - sub-control01_dwi.bvec - fmap/ - sub-control01_phasediff.nii.gz - sub-control01_phasediff.json - sub-control01_magnitude1.nii.gz -code/ - deface.py -derivatives/ -README -participants.tsv -dataset_description.json -CHANGES -``` +{{ MACROS___make_filetree_example( + { + "sub-control01": { + "anat":{ + "sub-control01_T1w.nii.gz": "", + "sub-control01_T1w.json": "", + "sub-control01_T2w.nii.gz": "", + "sub-control01_T2w.json": "", + }, + "func":{ + "sub-control01_task-nback_bold.nii.gz": "", + "sub-control01_task-nback_bold.json": "", + "sub-control01_task-nback_events.tsv": "", + "sub-control01_task-nback_physio.tsv.gz": "", + "sub-control01_task-nback_physio.json": "", + "sub-control01_task-nback_sbref.nii.gz": "", + }, + "dwi":{ + "sub-control01_dwi.nii.gz": "", + "sub-control01_dwi.bval": "", + "sub-control01_dwi.bvec": "", + }, + "fmap":{ + "sub-control01_phasediff.nii.gz": "", + "sub-control01_phasediff.json": "", + "sub-control01_magnitude1.nii.gz": "", + } + }, + "code": { + "deface.py": "" + }, + "derivatives": {}, + "README": "", + "participants.tsv": "", + "dataset_description.json": "", + "CHANGES": "", + } +) }} ## Unspecified data diff --git a/src/04-modality-specific-files/01-magnetic-resonance-imaging-data.md b/src/04-modality-specific-files/01-magnetic-resonance-imaging-data.md index bee7d711ca..87b2fb42f7 100644 --- a/src/04-modality-specific-files/01-magnetic-resonance-imaging-data.md +++ b/src/04-modality-specific-files/01-magnetic-resonance-imaging-data.md @@ -272,19 +272,24 @@ which are typically used in `part-mag`/`part-phase` or `part-real`/`part-imag` pairs of files. For example: -```Text -sub-01_part-mag_T1w.nii.gz -sub-01_part-mag_T1w.json -sub-01_part-phase_T1w.nii.gz -sub-01_part-phase_T1w.json -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "anat": { + "sub-01_part-mag_T1w.nii.gz": "", + "sub-01_part-mag_T1w.json": "", + "sub-01_part-phase_T1w.nii.gz": "", + "sub-01_part-phase_T1w.json": "", + }, + } + } +) }} Phase images MAY be in radians or in arbitrary units. The sidecar JSON file MUST include the units of the `phase` image. The possible options are `rad` or `arbitrary`. -For example: -sub-01_part-phase_T1w.json +For example, for `sub-01_part-phase_T1w.json`: ```Text { @@ -420,16 +425,20 @@ for more information on `dir` field specification. Multi-echo data MUST be split into one file per echo using the [`echo-`](../99-appendices/09-entities.md#echo) key-value pair. For example: -```Text -sub-01/ - func/ - sub-01_task-cuedSGT_run-1_echo-1_bold.nii.gz - sub-01_task-cuedSGT_run-1_echo-1_bold.json - sub-01_task-cuedSGT_run-1_echo-2_bold.nii.gz - sub-01_task-cuedSGT_run-1_echo-2_bold.json - sub-01_task-cuedSGT_run-1_echo-3_bold.nii.gz - sub-01_task-cuedSGT_run-1_echo-3_bold.json -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "func": { + "sub-01_task-cuedSGT_run-1_echo-1_bold.nii.gz": "", + "sub-01_task-cuedSGT_run-1_echo-1_bold.json": "", + "sub-01_task-cuedSGT_run-1_echo-2_bold.nii.gz": "", + "sub-01_task-cuedSGT_run-1_echo-2_bold.json": "", + "sub-01_task-cuedSGT_run-1_echo-3_bold.nii.gz": "", + "sub-01_task-cuedSGT_run-1_echo-3_bold.json": "", + }, + } + } +) }} Please note that the `` denotes the number/index (in the form of a nonnegative integer) of the echo not the echo time value which needs to be stored in the @@ -443,20 +452,25 @@ from the specification in the next major release. For backwards compatibility, `_phase` is considered equivalent to `_part-phase_bold`. When the `_phase` suffix is not used, each file shares the same name with the exception of the `part-` or `part-` key/value. + For example: -```Text -sub-01/ - func/ - sub-01_task-cuedSGT_part-mag_bold.nii.gz - sub-01_task-cuedSGT_part-mag_bold.json - sub-01_task-cuedSGT_part-phase_bold.nii.gz - sub-01_task-cuedSGT_part-phase_bold.json - sub-01_task-cuedSGT_part-mag_sbref.nii.gz - sub-01_task-cuedSGT_part-mag_sbref.json - sub-01_task-cuedSGT_part-phase_sbref.nii.gz - sub-01_task-cuedSGT_part-phase_sbref.json -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "func": { + "sub-01_task-cuedSGT_part-mag_bold.nii.gz": "", + "sub-01_task-cuedSGT_part-mag_bold.json": "", + "sub-01_task-cuedSGT_part-phase_bold.nii.gz": "", + "sub-01_task-cuedSGT_part-phase_bold.json": "", + "sub-01_task-cuedSGT_part-mag_sbref.nii.gz": "", + "sub-01_task-cuedSGT_part-mag_sbref.json": "", + "sub-01_task-cuedSGT_part-phase_sbref.nii.gz": "", + "sub-01_task-cuedSGT_part-phase_sbref.json": "", + }, + }, + } +) }} Some meta information about the acquisition MUST be provided in an additional JSON file. @@ -525,11 +539,15 @@ additional terms and their definitions. Example: -```Text -sub-control01/ - func/ - sub-control01_task-nback_bold.json -``` +{{ MACROS___make_filetree_example( + { + "sub-01": { + "func": { + "sub-control01_task-nback_bold.json": "", + }, + } + } +) }} ```JSON { @@ -679,50 +697,66 @@ two runs each, and the intent of the researcher is that all of them are part of a unique multipart scan, then they will tag all four runs with the same `MultipartID` (shown at the right-hand side of the file listing): -```Text -sub-