Write run results to disk (#829) #904
Conversation
| def default(self, obj): | ||
| if isinstance(obj, Decimal): | ||
| return float(obj) | ||
| return super(JSONEncoder, self).default(obj) |
There was a problem hiding this comment.
I really don't like the way Python handles JSON encoding. This looks like a good start in handling this better in dbt.
| fail = named_property('fail', 'True if this was a test and it failed') | ||
| status = named_property('status', 'The status of the model execution') | ||
| execution_time = named_property('execution_time', | ||
| 'The time in seconds to execute the model') |
dbt/compat.py
Outdated
|
|
||
| def write_json(path, data, **kwargs): | ||
| with _open(path, 'w') as fp: | ||
| json.dump(data, fp, **kwargs) |
There was a problem hiding this comment.
i don't love this as part of the public API of dbt.compat, but i see the need for it. is there an advantage to doing it this way, vs calling write_file with json.dumps(...) from dbt.clients.system?
There was a problem hiding this comment.
Performance is different. json.dump() uses less memory because it streams, but can take longer to run because each 'chunk' of json becomes a file.write() call and it bypasses the "one shot" encoder in C. I don't know which is faster in any given situation, I'll just change it to use json.dumps() and hope we never run out of memory, we'll probably gain a lot from using the good encoder anyway.
| @@ -4,6 +4,14 @@ | |||
|
|
|||
There was a problem hiding this comment.
this is going to have some rough conflicts w #883
There was a problem hiding this comment.
The merge conflicts will look horrible for sure, but looking at that PR they'll be pretty easy to resolve manually if I rebase on top of that after merge - I really only have to add a couple fields here and there.
dbt/compat.py
Outdated
| else: | ||
| with open(path, 'w') as f: | ||
| return f.write(to_string(s)) | ||
| return open(path, 'w') |
There was a problem hiding this comment.
i think you want return open(path, mode)
beckjake
force-pushed
the
write-run-results
branch
from
2a8929c
to
d05173d
Compare
beckjake
force-pushed
the
write-run-results
branch
from
d05173d
to
5ccaf5b
Compare
Per discussion, we'll be writing the run results to a separate file from the manifest itself. The run results include the compiled node, so the compiled sql is in there.
The actual code changes are small, the tests are just very verbose.