Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Add a dbt flag to specify the working directory #1544

Closed
heisencoder opened this issue Jun 14, 2019 · 11 comments
Closed

Feature request: Add a dbt flag to specify the working directory #1544

heisencoder opened this issue Jun 14, 2019 · 11 comments

Comments

@heisencoder
Copy link
Contributor

Feature

Feature description

Add a new flag to dbt to allow specifying the directory of the dbt_project.yml file (which in turn defines the relative directories of the other project files). This would be analogous to the -C and --git-dir flags for the git command. This would also complement the existing --profiles-dir flag.

Suggested name: --working-dir

Who will this benefit?

It would make tooling easier if it wasn't necessary to change the working directory before running dbt. For example, this reference implementation that combines dbt with airflow needs to include a 'cd' command within each BashOperator: https://gist.github.com/adamhaney/916a21b0adcabef4038c38e3ac96a36f
With this change, it would be sufficient just to pass this flag each time.

Also see slack discussion: https://getdbt.slack.com/messages/CBSQTAPLG/convo/CBSQTAPLG-1560546908.162800/

@heisencoder
Copy link
Contributor Author

I took a quick crack at this implementation, and it's a pretty straightforward modification to main.py in the latest release (0.11.1). However, this change runs into major merge conflicts with 1a0df17?diff=split when merged to dev/wilt-chamberlain, so I'm going to abandon this change for now (until wilt-chamberlain is released).

Also, I think a better flag name is --project-dir instead of --working-dir, since this is the directory for the project.

@beckjake
Copy link
Contributor

Do you mean 0.13.1 when you say latest release?

I am not really surprised by that, that was a heck of a merge.

Fortunately, I think the place to do this is 1) off wilt-chamberlain (or louisa-may-alcott once I merge in the current state of wilt-chamberlain) 2) in the run_from_args function, so maybe not too bad? Just look at parsed and issue an os.chdir into parsed.project_dir if it's specified, I assume...

@heisencoder
Copy link
Contributor Author

Yup! I meant 0.13.1. I'll take a quick look at run_from_args to see if that's easy in wilt-chamberlain.

@beckjake
Copy link
Contributor

Ok. it's possible I have overlooked something and the real place to do it is handle_and_check (the caller of run_from_args), right after parse_args, but I don't think anything there cares about your current working directory.

@heisencoder
Copy link
Contributor Author

The change I was planning against 0.13.1's main.py looked basically like this:

# (Define new flag named 'project-dir' and make it optional.)

def get_nearest_project_dir(parsed):
    # If the user provides an explicit project directory, use that.
    if parsed.project_dir:
        project_file = os.path.join(parsed.project_dir, "dbt_project.yml")
        if os.path.exists(project_file):
            return parsed.project_dir
        else:
            return None

    # Otherwise, use the current working directory and all its parents.
    cwd = os.getcwd()

    root_path = os.path.abspath(os.sep)
    while cwd != root_path:
        project_file = os.path.join(cwd, "dbt_project.yml")
        if os.path.exists(project_file):
            return cwd
        cwd = os.path.dirname(cwd)

    return None

So, wherever the 'get_nearest_project_dir' moved to, that's where I was thinking it would go. I didn't see anything in the new main.py that did an os.chdir(nearest_project_dir), so I'm assuming that got moved into a different file (or is now done via a different mechanism).

@beckjake
Copy link
Contributor

beckjake commented Jun 17, 2019

Oh, I see! That's reasonable enough, I hadn't thought of that. It should now be in core/task/base.py, you'll probably have to change move_to_nearest_project_dir and it to take arguments again, which is fine! The interface I think we care about preserving is BaseTask.from_args, anything under that is just implementation details.

In that method, args is a parsed argparse.Namespace, so all the same ideas should apply.

@heisencoder
Copy link
Contributor Author

Good to know! That should make it a straightforward change then.

@heisencoder
Copy link
Contributor Author

Hi @beckjake, where should I put the unit test for this change?

@beckjake
Copy link
Contributor

beckjake commented Jun 17, 2019

There's a set of tests in test/unit/test_config.py that deal with this sort of thing, in particular TestProjectFile and TestRuntimeConfigFiles. You'll have to update the Args object defined in there but I kind of assume you'll have to do that anyway.

@heisencoder
Copy link
Contributor Author

Hi @beckjake: Let me know if the unit test looks good. I couldn't get TestProjectFile and TestRuntimeConfigFiles to easily invoke the modified code, so I added a new test case called TestRunOperationTask to create a new RunOperationTask and make sure the os directory was appropriately changed. If the test looks good, this change is ready for merging.

@drewbanin
Copy link
Contributor

closed by #1549

@drewbanin drewbanin added this to the Wilt Chamberlain milestone Jun 19, 2019
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
In particular, this is the directory that contains the dbt_project.yml
file and all the related project files and directories.

Without this flag, it is necessary to cd to the project directory before
running dbt.

See #1544


automatic commit by git-black, original commits:
  2e7c1fd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants