Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download datasets #162

Merged
merged 38 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
36da236
Initial commit
martwo Jul 19, 2023
4839fd5
Add config option to download dataset from origin
martwo Jul 20, 2023
a848f9f
Complete method
martwo Jul 20, 2023
0333305
Make iminuit an optional tool
martwo Jul 20, 2023
d1d5530
Add some print debugging
martwo Jul 20, 2023
8266a95
Load the base data first
martwo Jul 20, 2023
bb40d79
Merge branch 'master' into download_datasets
martwo Jul 20, 2023
339cbd3
Add pretty string and logger debug messages
martwo Jul 20, 2023
fd91a99
Add doc string
martwo Jul 20, 2023
10aab73
Create symlink if origin is available locally
martwo Jul 21, 2023
0089304
Add is_directory flag
martwo Jul 21, 2023
cd84014
Add properties to DatasetOrigin
martwo Jul 21, 2023
2d8e8e0
Implement file mode feature
martwo Jul 21, 2023
4485140
Transfer individual dataset files
martwo Jul 23, 2023
237c0f6
Use new DatasetOrigin interface, ensure directories exist
martwo Jul 23, 2023
f4c0b58
Allow None for post_transfer_func
martwo Jul 29, 2023
d695a65
Rename user to username
martwo Jul 29, 2023
7b05f95
Support port in wget
martwo Jul 29, 2023
a6dfb2f
Create local sub dirs
martwo Jul 29, 2023
087dc5e
Add convenient method
martwo Jul 29, 2023
80eb22d
Merge branch 'master' into download_datasets
martwo Jul 30, 2023
88a6533
Skip existing files
martwo Jul 30, 2023
b4fba84
Skip existing files
martwo Jul 30, 2023
f1fcfdb
Add TestData dataset definition
martwo Jul 31, 2023
380a0f3
Rename method
martwo Jul 31, 2023
16067f4
Improve dataset handling and error handling of data transfers
martwo Jul 31, 2023
cd8a2c8
Add unit test for rsync dataset transfers
martwo Jul 31, 2023
9f3a344
Add unit test for wget dataset transfer
martwo Aug 1, 2023
d63e778
Fix origin
martwo Aug 1, 2023
f9780ea
Make DatasetTransfer.transfer an instance method
martwo Aug 1, 2023
e9c04af
protocol is not part of origin
martwo Aug 4, 2023
fdc250a
Add docs for dataset
martwo Aug 4, 2023
71e5d56
Fix typo
martwo Aug 4, 2023
a8e00af
Merge branch 'master' into download_datasets
martwo Aug 7, 2023
9ad0cb7
Add changelog info
martwo Aug 7, 2023
bedfcc5
Merge branch 'master' into download_datasets
martwo Aug 8, 2023
fdeb175
Remove obsolete method
martwo Aug 8, 2023
0bb71d7
Improve doc string
martwo Aug 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/pythonpackage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ jobs:
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with unittest
- name: Run tests with unittest
env:
ICECUBE_PASSWORD: ${{ secrets.ICECUBE_PASSWORD }}
run: |
./tests/run.sh
13 changes: 13 additions & 0 deletions CHANGELOG.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
This file contains a log-book for major changes between releases.

v23.2.1
=======
- Allow the definition of an origin of a dataset via the
core.dataset.DatasetOrigin class and download the dataset automatically from
the origin to the local host. The following transfer methods are provided:

- wget
- rsync

Unit tests for these transfer methods are added.

- The iminuit package is an optional tool now.

v23.2.0
=======
- Complete overhaul of SkyLLH for more generic handling of parameters
Expand Down
Loading