-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: --fail-fast argument for dbt run and dbt test #1649
Comments
Thanks @JasonGluck! I think we might have built this at a hackathon once, but I don't think the code ever made it into prod :/ I think the operative line of code is right here. I imagine we could add some logic there which consulted the value of the Is this something you'd be interested in contributing a PR for? |
Dupe of #871 - closing that one. |
This would save our team a lot of time! |
I'm currently thinking about the behavior of Now I'm looking for place where we can gentle raise in-between testing. Of course I really appreciate any help with that 🍺 My current state of work could be found at: raalsky/fail-fast I'm also thinking about testing these parts of codebase in some sort of providing a desired ordered list of nodes to Of course If @sjwhitworth is still working on it just let me know 😉 |
What a great point! That's definitely a hole in the process. I think the key issue is using
We could do this (but probably abstract it so we aren't reaching so far into internals):
I would probably do that by writing a |
Hi @drewbanin and @beckjake |
@beckjake with self.job_queue.inner.all_tasks_done:
while self.job_queue.inner.unfinished_tasks:
tell_the_world_is_ending()
self.job_queue.inner.all_tasks_done.wait() I think the def task_done(self):
'''Indicate that a formerly enqueued task is complete.
Used by Queue consumer threads. For each get() used to fetch a task,
a subsequent call to task_done() tells the queue that the processing
on the task is complete.
If a join() is currently blocking, it will resume when all items
have been processed (meaning that a task_done() call was received
for every item that had been put() into the queue).
Raises a ValueError if called more times than there were items
placed in the queue.
'''
with self.all_tasks_done:
unfinished = self.unfinished_tasks - 1
if unfinished <= 0:
if unfinished < 0:
raise ValueError('task_done() called too many times')
self.all_tasks_done.notify_all()
self.unfinished_tasks = unfinished We're blocking on I think we should consider also rewrite of while not nothing_to_be_processed:
node = self.job_queue.get()
self._raise_set_error()
runner = self.get_runner(node)
...
self._submit(pool, args, callback)
# Only an approximation of queue size!
if self.job_queue.qsize() >= self.config.threads:
self.job_queue.wait_until_something_was_done() |
Keep in mind it's not only about |
I've pushed an MVP to Raalsky/fail-fast (need a lot in carrying about concurrency 😭 ) but it works pretty close to what I've expected. |
Oh, I didn't realize that |
Feature
--fail-fast
argument fordbt run
anddbt test
Feature description
An optional argument,
--fail-fast
, for dbt run and dbt test which will stop model builds or test runs upon a single failure. It may also be worth adding a parameter which will define the amount of failures to allow before stopping.Who will this benefit?
All dbt users. Will be useful during the development process. Can also help reduce the amount of queries to the database.
Questions
How should this work with test severity?
The text was updated successfully, but these errors were encountered: