Data adapters #128
Merged
Data adapters #128
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elijahbenizzy
force-pushed
the
data-adapters
branch
10 times, most recently
from
d98ec4d to
75fa69f
Compare
See #46 for some initial context. Basic design is: 1. A decorator that loads data and injects into a function 2. A set of Data Loaders that can be constructed from kwargs (they're all dataclasses) 3. A LoadFromDecorator that takes in a variety of data loader classes that are possible to choose from 4. A registry that specifies a mapping of key -> data loader classes, allowing us to polymorphically select them More todos, up next 1. Add tests for the default loaders 2. Add safeguards to ensure any loader is a dataclass 3. Add documentation We also add a register_adapter function in the registry. Note that this just moves the adapters we currently have around, and does not add new ones.
Also moves some files to more convenient, central locations
We expect to add more optional arguments over time, but these should get people started.
This makes data loaders specify the types they apply to, rather than filling out the applies_to function. This way, we can easily document which data loaders are used for which types, to make it easier to clarify
Still don't have SQL support yet, but that's coming soon.
elijahbenizzy
force-pushed
the
data-adapters
branch
from
993cd1c to
aa8ba68
Compare
elijahbenizzy
force-pushed
the
data-adapters
branch
2 times, most recently
from
dffb1f3 to
097ea28
Compare
This decorates the output of a function -- creating a node to append to the end that saves it. This has similar functionality to data loaders, and has been grouped togther under the data loading umbrella.
elijahbenizzy
force-pushed
the
data-adapters
branch
from
097ea28 to
dde909f
Compare
elijahbenizzy
force-pushed
the
data-adapters
branch
from
f87311f to
8bfb284
Compare
skrawcz
reviewed
Apr 4, 2023
skrawcz
reviewed
Apr 6, 2023
elijahbenizzy
force-pushed
the
data-adapters
branch
from
8bdbc5b to
fd5614f
Compare
elijahbenizzy
force-pushed
the
data-adapters
branch
from
fd5614f to
2bcb8b1
Compare
This allows us to render them specially in a viz or do a query for data sources.
elijahbenizzy
force-pushed
the
data-adapters
branch
from
2bcb8b1 to
1a30446
Compare
For the save_to decorator. Since I think output is more consistent with our language. We really have not been consistent in the driver/code base at all. Choosing `output` because the `.execute()` function requests final_vars and is documented as the `outputs` one wants. Thus I think reading `output_name_` will help people identify that this is what they would use if they wanted to get the results out.
skrawcz
approved these changes
Apr 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Short description explaining the high-level reason for the pull request]
Changes
How I tested this
Notes
Checklist