Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allows the profile directory to be set with an environment var #1049

Merged
merged 3 commits into from
Oct 17, 2018
Merged

Allows the profile directory to be set with an environment var #1049

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
45bbaf2
Allows the profile directory to be set with an environment var
mikekaminsky Oct 9, 2018
a54b5e3
Shorter (and more pythonic) global variable setting
mikekaminsky Oct 12, 2018
e0de86e
Fix line length. Thanks Pep8!
mikekaminsky Oct 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,6 @@ target/
*.sublime-*

.python-version

# Vim
*.sw*
11 changes: 8 additions & 3 deletions dbt/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os.path
import os
from copy import deepcopy
import hashlib
import pprint
Expand All @@ -23,6 +24,10 @@
DEFAULT_USE_COLORS = True
DEFAULT_PROFILES_DIR = os.path.join(os.path.expanduser('~'), '.dbt')

if os.environ.get('DBT_PROFILES_DIR') is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the way this code works, the supplied env var won't have the ~ expanded. That means that quoted vars won't be expanded, whereas unquoted vars will be expanded.

$ DBT_PROFILES_DIR="~/.dbt/profiles.yml"
$ echo $DBT_PROFILES_DIR
~/.dbt/profiles.yml

$ DBT_PROFILES_DIR=~/.dbt/profiles.yml
$ echo $DBT_PROFILES_DIR
/Users/drew/.dbt/profiles.yml

In the former case, I think dbt will choke on the tilde. Maybe that's just caveat emptor, and a certain amount of bash knowledge is required to use env vars anyway. Alternatively, we could do something like:

if os.environ.get('DBT_PROFILES_DIR') is not None:
  PROFILES_DIR = os.path.expanduser(os.environ.get('DBT_PROFILES_DIR'))

I don't feel super strongly about this, but wanted to raise the topic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good point! Combining our two suggestions:
PROFILES_DIR = os.path.expanduser(os.environ.get('DBT_PROFILES_DIR', DEFAULT_PROFILES_DIR))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ that's a little funky because we'll be expanding the tilde twice when no env var is present, but I don't think it would be an issue, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only way it would be possible to have a problem is if there were a posix system where the home directory started with a literal ~, in which case the user will have bigger problems.

PROFILES_DIR = os.environ.get('DBT_PROFILES_DIR')
else:
PROFILES_DIR = DEFAULT_PROFILES_DIR
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could just be PROFILES_DIR = os.environ.get('DBT_PROFILES_DIR', DEFAULT_PROFILES_DIR)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

D'oh! Will fix.


INVALID_PROFILE_MESSAGE = """
dbt encountered an error while trying to read your profiles.yml file.
Expand All @@ -41,7 +46,7 @@
defined in your profiles.yml file. You can find profiles.yml here:

{profiles_file}/profiles.yml
""".format(profiles_file=DEFAULT_PROFILES_DIR)
""".format(profiles_file=PROFILES_DIR)


def read_profile(profiles_dir):
Expand All @@ -62,7 +67,7 @@ def read_profile(profiles_dir):
def read_profiles(profiles_dir=None):
"""This is only used in main, for some error handling"""
if profiles_dir is None:
profiles_dir = DEFAULT_PROFILES_DIR
profiles_dir = PROFILES_DIR

raw_profiles = read_profile(profiles_dir)

Expand Down Expand Up @@ -624,7 +629,7 @@ def from_args(cls, args, project_profile_name=None, cli_vars=None):

threads_override = getattr(args, 'threads', None)
# TODO(jeb): is it even possible for this to not be set?
profiles_dir = getattr(args, 'profiles_dir', DEFAULT_PROFILES_DIR)
profiles_dir = getattr(args, 'profiles_dir', PROFILES_DIR)
target_override = getattr(args, 'target', None)
raw_profiles = read_profile(profiles_dir)
profile_name = cls._pick_profile_name(args.profile,
Expand Down
6 changes: 3 additions & 3 deletions dbt/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@

from dbt.utils import ExitCodes
from dbt.config import Project, RuntimeConfig, DbtProjectError, \
DbtProfileError, DEFAULT_PROFILES_DIR, read_config, \
DbtProfileError, PROFILES_DIR, read_config, \
send_anonymous_usage_stats, colorize_output, read_profiles
from dbt.exceptions import DbtProfileError, DbtProfileError, RuntimeException

Expand Down Expand Up @@ -298,11 +298,11 @@ def parse_args(args):

base_subparser.add_argument(
'--profiles-dir',
default=DEFAULT_PROFILES_DIR,
default=PROFILES_DIR,
type=str,
help="""
Which directory to look in for the profiles.yml file. Default = {}
""".format(DEFAULT_PROFILES_DIR)
""".format(PROFILES_DIR)
)

base_subparser.add_argument(
Expand Down
2 changes: 1 addition & 1 deletion dbt/task/debug.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
class DebugTask(BaseTask):
def path_info(self):
open_cmd = dbt.clients.system.open_dir_cmd()
profiles_dir = dbt.config.DEFAULT_PROFILES_DIR
profiles_dir = dbt.config.PROFILES_DIR

message = PROFILE_DIR_MESSAGE.format(
open_cmd=open_cmd,
Expand Down
2 changes: 1 addition & 1 deletion dbt/task/init.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def get_addendum(self, project_name, profiles_path):
def run(self):
project_dir = self.args.project_name

profiles_dir = dbt.config.DEFAULT_PROFILES_DIR
profiles_dir = dbt.config.PROFILES_DIR
profiles_file = os.path.join(profiles_dir, 'profiles.yml')

self.create_profiles_dir(profiles_dir)
Expand Down