Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Downloader utility class to use static dask cluster #1161

Merged
merged 23 commits into from
Sep 22, 2023
Merged
Changes from 4 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
08e2996
update downloader to be static variable
efajardo-nv Aug 29, 2023
e859592
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Aug 29, 2023
5bd447e
Merge branch 'branch-23.11' of https://github.com/nv-morpheus/Morpheu…
efajardo-nv Aug 29, 2023
9e56ddd
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Aug 30, 2023
1ecd371
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Aug 30, 2023
2b79fd6
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Sep 1, 2023
ff36997
Merge branch 'branch-23.11' of https://github.com/nv-morpheus/Morpheu…
efajardo-nv Sep 11, 2023
46bbe5b
make dask cluster static instead per feedback
efajardo-nv Sep 11, 2023
a98b0ea
Merge branch 'file-to-df-loader-fix' of https://github.com/efajardo-n…
efajardo-nv Sep 11, 2023
e40af22
get_dask_cluster update
efajardo-nv Sep 11, 2023
45d9d8e
Merge branch 'branch-23.11' of https://github.com/nv-morpheus/Morpheu…
efajardo-nv Sep 12, 2023
53b967b
update downloader unit tests
efajardo-nv Sep 12, 2023
54fc756
add comment
efajardo-nv Sep 12, 2023
18add2e
update test_dfp_file_to_df
efajardo-nv Sep 12, 2023
52f3703
fix module dfp cmd example
efajardo-nv Sep 13, 2023
c485ded
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Sep 13, 2023
44c1f4c
create second downloader in test_get_dask_cluster
efajardo-nv Sep 13, 2023
ed8cfc4
Merge branch 'branch-23.11' of https://github.com/nv-morpheus/Morpheu…
efajardo-nv Sep 18, 2023
e874961
use mutex when getting dask cluster
efajardo-nv Sep 18, 2023
4fa8d12
mutex update
efajardo-nv Sep 18, 2023
9cb2eb8
style fixes
efajardo-nv Sep 18, 2023
f08b27a
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Sep 19, 2023
356d79a
Merge branch 'branch-23.11' into file-to-df-loader-fix
efajardo-nv Sep 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions morpheus/loaders/file_to_df_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ def file_to_df_loader(control_message: ControlMessage, task: dict):
parser_kwargs = config.get("parser_kwargs", None)
cache_dir = config.get("cache_dir", None)

downloader = Downloader()
if not hasattr(file_to_df_loader, "downloader"):
file_to_df_loader.downloader = Downloader()
efajardo-nv marked this conversation as resolved.
Show resolved Hide resolved

if (cache_dir is None):
cache_dir = "./.cache"
Expand Down Expand Up @@ -174,7 +175,7 @@ def get_or_create_dataframe_from_s3_batch(file_name_batch: typing.List[str]) ->

# Loop over dataframes and concat into one
try:
dfs = downloader.download(download_buckets, download_method_func)
dfs = file_to_df_loader.downloader.download(download_buckets, download_method_func)
except Exception:
logger.exception("Failed to download logs. Error: ", exc_info=True)
raise
Expand Down