Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dandi-cli readme edit #1170

Merged
merged 4 commits into from
Dec 12, 2022
Merged

dandi-cli readme edit #1170

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3d3bb9a
dandi-cli readme edit
melster1010 Dec 1, 2022
79c829c
fix spelling
melster1010 Dec 6, 2022
87c37b0
Incorporate review feedback
melster1010 Dec 7, 2022
97555c7
Update dandi and ls --help output
melster1010 Dec 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 73 additions & 32 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,36 @@
[![PyPI version fury.io](https://badge.fury.io/py/dandi.svg)](https://pypi.python.org/pypi/dandi/)
[![Documentation Status](https://readthedocs.org/projects/dandi/badge/?version=latest)](https://dandi.readthedocs.io/en/latest/?badge=latest)

This project is under heavy development. Beware of [hidden](I-wish-we-knew) and
[disclosed](https://github.com/dandi/dandi-cli/issues) issues, or
The [DANDI Python client](https://pypi.org/project/dandi/) allows you to:

* Download `Dandisets` and individual subject folders or files
* Validate data to locally conform to standards
* Organize your data locally before upload
* Upload `Dandisets`
melster1010 marked this conversation as resolved.
Show resolved Hide resolved
* Interact with the DANDI archive's web API from Python
* Delete data in the DANDI archive
* Perform other auxiliary operations with data or the DANDI archive

**Note**: This project is under heavy development. See [the issues log](https://github.com/dandi/dandi-cli/issues) or
[Work-in-Progress (WiP)](https://github.com/dandi/dandi-cli/pulls).

## Installation

At the moment DANDI client releases are [available from PyPI](https://pypi.org/project/dandi)
and [conda-forge](https://anaconda.org/conda-forge/dandi). You could
install them in your Python (native, virtualenv, or conda) environment via
DANDI Client releases are [available from PyPI](https://pypi.org/project/dandi)
and [conda-forge](https://anaconda.org/conda-forge/dandi). Install them in your Python (native, virtualenv, or
conda) environment via

pip install dandi

or

conda install -c conda-forge dandi

if you are in a conda environment.

## dandi tool
## CLI Tool

This package provides a `dandi` command line utility with a basic interface
which should assist you in preparing and uploading your data to and/or obtaining
data from the http://dandiarchive.org:
This package provides a command line utility with a basic interface
to help you prepare and upload your data to, or obtain data from, the [DANDI archive](http://dandiarchive.org):

```bash
$> dandi
Expand All @@ -45,51 +52,85 @@ Usage: dandi [OPTIONS] COMMAND [ARGS]...

Options:
--version
-l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Log level name [default: INFO]
-l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
Log level (case insensitive). May be
specified as an integer. [default: INFO]
--pdb Fall into pdb if errors out
--help Show this message and exit.

Commands:
download Download a file or entire folder from DANDI
ls List .nwb files and dandisets metadata.
organize (Re)organize files according to the metadata.
register Register a new dandiset in the DANDI archive
upload Upload dandiset (files) to DANDI archive.
validate Validate files for NWB (and DANDI) compliance.
delete Delete dandisets and assets from the server.
digest Calculate file digests
download Download a file or entire folder from DANDI.
instances List known Dandi Archive instances that the CLI can...
ls List .nwb files and dandisets metadata.
move Move or rename assets in a local Dandiset and/or on...
organize (Re)organize files according to the metadata.
service-scripts Various utility operations
shell-completion Emit shell script for enabling command completion.
upload Upload Dandiset files to DANDI Archive.
validate Validate files for NWB and DANDI compliance.
validate-bids Validate BIDS paths.
```

Each of the commands has a set of options to alter their behavior. Please run
`dandi COMMAND --help` to get more information, e.g.
Each of the commands has a set of options to alter its behavior. Run
`dandi COMMAND --help` to get more information:

```
$> dandi ls --help
Usage: dandi ls [OPTIONS] [PATHS]...

List .nwb files metadata
Usage: dandi ls [OPTIONS] PATH|URL

List .nwb files and dandisets metadata.

The arguments may be either resource identifiers or paths to local
files/directories.

Accepted resource identifier patterns:
- DANDI:<dandiset id>[/<version>]
- https://dandiarchive.org/...
- https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')
- https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]
- https://*dandiarchive-org.netflify.app/...
- https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]
- https://<server>[/api]/assets/<asset id>[/download]
- https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]
- https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>
- dandi://<instance name>/<dandiset id>[@<version>][/<path>]
- https://<server>/...

Options:
-F, --fields TEXT Comma-separated list of fields to display.
An empty value to trigger a list of
available fields to be printed out
-f, --format [auto|pyout|json|json_pp|yaml]
-f, --format [auto|pyout|json|json_pp|json_lines|yaml]
Choose the format/frontend for output. If
'auto', 'pyout' will be used in case of
multiple files, and 'yaml' for a single
file.
-r, --recursive Recurse into content of
dandisets/directories. Only .nwb files will
be considered.
-J, --jobs INTEGER Number of parallel download jobs. [default:
6]
--metadata [api|all|assets]
--schema VERSION Convert metadata to new schema version
--help Show this message and exit.
```
Comment on lines -48 to 118
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be out of scope for this PR, but this section seems like it can be simplified to letting the user know that they can run dandi --help and dandi <subcommand> --help to get manpages. Listing sample output means we'll need to keep updating this section every time the CLI changes its interface.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That had crossed my mind too, wasn't sure how often this changed. Added to my Phase 2 docs backlog for 2023.


See [DANDI Handbook](https://www.dandiarchive.org/handbook/10_using_dandi/)
for examples on how to use this client in various use cases.

## Development/contributing
## Third-party Components

Please see [DEVELOPMENT.md](./DEVELOPMENT.md) file.
**dandi/tests/skip.py** -- from https://github.com/ReproNim/reproman, as of v0.2.1-40-gf4f026d
Copyright (c) 2016-2020 ReproMan Team
Comment on lines -86 to +124
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also out of scope for this PR, but I think this section can be struck. I don't think it adds any useful information about DANDI CLI.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to my Phase 2 docs backlog for 2023.


## 3rd party components included
## Resources

### dandi/tests/skip.py
* To learn how to interact with the DANDI archive and for examples on how to use the DANDI Client in various use cases,
see [the handbook](https://www.dandiarchive.org/handbook/).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be useful to highlight sections of the handbook specifically related to the CLI:

Suggested change
see [the handbook](https://www.dandiarchive.org/handbook/).
see [the handbook](https://www.dandiarchive.org/handbook/),
specifically the sections on using the CLI to
[download](https://www.dandiarchive.org/handbook/12_download/) and
[upload](https://www.dandiarchive.org/handbook/13_upload/) `Dandisets`.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that specific guidance would be useful. Added to my Phase 2 docs backlog for 2023.


From https://github.com/ReproNim/reproman, as of v0.2.1-40-gf4f026d
Copyright (c) 2016-2020 ReproMan Team
* To get help:
- ask a question: https://github.com/dandi/helpdesk/discussions
- file a feature request or bug report: https://github.com/dandi/helpdesk/issues/new/choose
- contact the DANDI team: help@dandiarchive.org

* To understand how to contribute to the dandi-cli repository, see the [DEVELOPMENT.md](./DEVELOPMENT.md) file.