Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX: Require environment variable to use DataLad #8

Merged
merged 3 commits into from
Mar 13, 2019
Merged

FIX: Require environment variable to use DataLad #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a076d53
FIX: Require environment variable to use DataLad
oesteban Mar 13, 2019
a653a49
revise how circle builds are configured
oesteban Mar 13, 2019
1bc9513
fix try..except logic
oesteban Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ jobs:
pyenv global 3.5.2
virtualenv venv
pip install -r /tmp/src/templateflow/requirements.txt
pip install datalad
pip install "setuptools>=27.0" twine

- save_cache:
Expand All @@ -48,6 +49,9 @@ jobs:

- run:
name: Run tests (w/o datalad)
environment:
TEMPLATEFLOW_USE_DATALAD: 0
TEMPLATEFLOW_HOME: "/tmp/data/templateflow"
command: |
pyenv global 3.5.2
virtualenv venv
Expand All @@ -56,19 +60,22 @@ jobs:

- run:
name: Run tests (pulling from S3)
environment:
TEMPLATEFLOW_USE_DATALAD: 0
TEMPLATEFLOW_HOME: "/tmp/data/templateflow-S3"
command: |
pyenv global 3.5.2
virtualenv venv
export TEMPLATEFLOW_HOME="/tmp/data/templateflow-S3"
pytest -vsx --doctest-modules /tmp/src/templateflow/templateflow

- run:
name: Run tests (w/ datalad)
environment:
TEMPLATEFLOW_USE_DATALAD: 1
command: |
unset TEMPLATEFLOW_HOME
pyenv global 3.5.2
virtualenv venv
pip install datalad
pytest -vsx --doctest-modules /tmp/src/templateflow/templateflow

- run:
Expand Down
15 changes: 12 additions & 3 deletions templateflow/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"""
from pathlib import Path
from json import loads
from .conf import TF_LAYOUT, TF_S3_ROOT
from .conf import TF_LAYOUT, TF_S3_ROOT, TF_USE_DATALAD


def get(template, **kwargs):
Expand All @@ -25,12 +25,21 @@ def get(template, **kwargs):
out_file = [Path(p) for p in TF_LAYOUT.get(
template=template, return_type='file', **kwargs)]

# Try plain URL fetch first
for filepath in [p for p in out_file
if p.is_file() and p.stat().st_size == 0]:
_s3_get(filepath)

for filepath in [p for p in out_file if not p.is_file()]:
_datalad_get(filepath)
if TF_USE_DATALAD:
for filepath in [p for p in out_file if not p.is_file()]:
_datalad_get(filepath)

not_fetched = [p for p in out_file
if not p.is_file() or p.stat().st_size == 0]

if any(not_fetched):
raise RuntimeError(
"Could not fetch template files: %s" % ', '.join(not_fetched))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be the error could include a message about TF_USE_DATALAD (if that one was not set), to instruct users about a possible way to get those files where applicable?


if len(out_file) == 1:
return out_file[0]
Expand Down
20 changes: 13 additions & 7 deletions templateflow/conf/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,26 +11,32 @@
TF_HOME = Path(getenv('TEMPLATEFLOW_HOME', str(TF_DEFAULT_HOME)))
TF_GITHUB_SOURCE = 'https://github.com/templateflow/templateflow.git'
TF_S3_ROOT = 'https://templateflow.s3.amazonaws.com'
TF_USE_DATALAD = getenv('TEMPLATEFLOW_USE_DATALAD', 'false').lower() in (
'true', 'on', '1')

_msg = """\
TemplateFlow: repository not found at %s. Populating a TemplateFlow stub.
If the path reported above is not the desired location for Templateflow, \
If the path reported above is not the desired location for TemplateFlow, \
please set the TEMPLATEFLOW_HOME environment variable.
""" % TF_HOME

if not TF_HOME.exists() or not list(TF_HOME.iterdir()):
warn(_msg, ResourceWarning)
try:
from datalad.api import install
except ImportError:
if TF_USE_DATALAD:
try:
from datalad.api import install
except ImportError:
TF_USE_DATALAD = False
else:
TF_HOME.parent.mkdir(exist_ok=True, parents=True)
install(path=str(TF_HOME), source=TF_GITHUB_SOURCE, recursive=True)

if not TF_USE_DATALAD:
from zipfile import ZipFile
TF_HOME.mkdir(exist_ok=True, parents=True)
with ZipFile(resource_filename('templateflow',
'conf/templateflow-skel.zip'), 'r') as zipref:
zipref.extractall(str(TF_HOME))
else:
TF_HOME.parent.mkdir(exist_ok=True, parents=True)
install(path=str(TF_HOME), source=TF_GITHUB_SOURCE, recursive=True)

TF_LAYOUT = Layout(
TF_HOME, validate=False, config='templateflow',
Expand Down