Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two data loaders can't share the same parameter #232

Closed
elijahbenizzy opened this issue Jul 25, 2023 · 3 comments
Closed

Two data loaders can't share the same parameter #232

elijahbenizzy opened this issue Jul 25, 2023 · 3 comments
Labels
decorators triage label for issues that need to be triaged.

Comments

@elijahbenizzy
Copy link
Collaborator

Short description explaining the high-level reason for the new issue.

Current behavior

This should work:

@load_from.csv(path=source("weekday_file"))
def weekday_data(data: pd.DataFrame) -> pd.DataFrame:
    return data


@load_from.csv(path=source("weekend_file"))
def weekend_data(data: pd.DataFrame) -> pd.DataFrame:
    return data

But it does not. This is sloppy -- it produces a node called data, which should have some namespace associated with it.

Stack Traces

Traceback (most recent call last):
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/examples/parallelism/run.py", line 17, in <module>
    main()
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/examples/parallelism/run.py", line 7, in main
    dr = driver.Builder(). \
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/driver.py", line 51, in wrapped_fn
    return call_fn(*args, **kwargs)
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/driver.py", line 954, in build
    return DriverV2(
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/driver.py", line 720, in __init__
    super(DriverV2, self).__init__(config, *modules)
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/driver.py", line 123, in __init__
    raise e
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/driver.py", line 118, in __init__
    self.graph = graph.FunctionGraph(*modules, config=config, adapter=adapter)
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/graph.py", line 221, in __init__
    self.nodes = create_function_graph(*modules, config=self._config, adapter=adapter)
  File "/Users/elijahbenizzy/dev/dagworks/os/hamilton/hamilton/graph.py", line 95, in create_function_graph
    raise ValueError(
ValueError: Cannot define function data more than once. Already defined by function <function weekend_data at 0x16c059310>```

# Expected behavior

Should not try to create a node called `data` -- `NodeInjector` should be smart enough to handle this with namespaces.


# Additional context
Add any other context about the problem here.
@elijahbenizzy elijahbenizzy added triage label for issues that need to be triaged. decorators labels Jul 25, 2023
@skrawcz
Copy link
Collaborator

skrawcz commented Aug 5, 2023

@elijahbenizzy was this fixed?

@elijahbenizzy
Copy link
Collaborator Author

@elijahbenizzy was this fixed?

Nope, this is on the queue

@elijahbenizzy
Copy link
Collaborator Author

This is now fixed and released with #264

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
decorators triage label for issues that need to be triaged.
Projects
None yet
Development

No branches or pull requests

2 participants