Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQL adapters #46

Closed
HamiltonRepoMigrationBot opened this issue Feb 26, 2023 · 3 comments
Closed

SQL adapters #46

HamiltonRepoMigrationBot opened this issue Feb 26, 2023 · 3 comments
Labels
migrated-from-old-repo Migrated from old repository product idea

Comments

@HamiltonRepoMigrationBot
Copy link
Collaborator

Issue by elijahbenizzy
Thursday Sep 15, 2022 at 03:45 GMT
Originally opened as stitchfix/hamilton#197

See design here: https://docs.google.com/document/d/1-xVPX7PqyENOk31bXdcAVVPYSHaGMokeFiR85d_zoOo/edit.
@HamiltonRepoMigrationBot
Copy link
Collaborator Author

Comment by elijahbenizzy
Friday Oct 28, 2022 at 22:19 GMT

TLDR -- an adapter that enables you to run either a transform or a load from sql. Framework can abstract away the executing/connecting piece. This would be especially nice with duckdb -- see stitchfix/hamilton#158.

@elijahbenizzy elijahbenizzy mentioned this issue Feb 26, 2023
Closed
@elijahbenizzy elijahbenizzy added the migrated-from-old-repo Migrated from old repository label Feb 26, 2023
elijahbenizzy added a commit that referenced this issue Apr 1, 2023
@elijahbenizzy
Adds implementation of data loaders
107bae9 
See #46 for some initial
context.

Basic design is:
1. A decorator that loads data and injects into a function
2. A set of Data Loaders that can be constructed from kwargs (they're
   all dataclasses)
3. A LoadFromDecorator that takes in a variety of data loader classes
   that are possible to choose from
4. A registry that specifies a mapping of key -> data loader classes,
   allowing us to polymorphically select them

More todos, up next
1. Add tests for the default loaders
2. Add safeguards to ensure any loader is a dataclass
3. Add documentation
elijahbenizzy added a commit that referenced this issue Apr 2, 2023
@elijahbenizzy
Adds implementation of data loaders
b277352 
See #46 for some initial
context.

Basic design is:
1. A decorator that loads data and injects into a function
2. A set of Data Loaders that can be constructed from kwargs (they're
   all dataclasses)
3. A LoadFromDecorator that takes in a variety of data loader classes
   that are possible to choose from
4. A registry that specifies a mapping of key -> data loader classes,
   allowing us to polymorphically select them

More todos, up next
1. Add tests for the default loaders
2. Add safeguards to ensure any loader is a dataclass
3. Add documentation
elijahbenizzy added a commit that referenced this issue Apr 2, 2023
@elijahbenizzy
Adds implementation of data loaders
4f8993f 
See #46 for some initial
context.

Basic design is:
1. A decorator that loads data and injects into a function
2. A set of Data Loaders that can be constructed from kwargs (they're
   all dataclasses)
3. A LoadFromDecorator that takes in a variety of data loader classes
   that are possible to choose from
4. A registry that specifies a mapping of key -> data loader classes,
   allowing us to polymorphically select them

More todos, up next
1. Add tests for the default loaders
2. Add safeguards to ensure any loader is a dataclass
3. Add documentation

We also add a register_adapter function in the registry. Note that this
just moves the adapters we currently have around, and does not add new
ones.
elijahbenizzy added a commit that referenced this issue Apr 3, 2023
@elijahbenizzy
Adds implementation of data loaders
96ac185 
See #46 for some initial
context.

Basic design is:
1. A decorator that loads data and injects into a function
2. A set of Data Loaders that can be constructed from kwargs (they're
   all dataclasses)
3. A LoadFromDecorator that takes in a variety of data loader classes
   that are possible to choose from
4. A registry that specifies a mapping of key -> data loader classes,
   allowing us to polymorphically select them

More todos, up next
1. Add tests for the default loaders
2. Add safeguards to ensure any loader is a dataclass
3. Add documentation

We also add a register_adapter function in the registry. Note that this
just moves the adapters we currently have around, and does not add new
ones.
elijahbenizzy added a commit that referenced this issue Apr 10, 2023
@elijahbenizzy
Adds implementation of data loaders
f4eafca 
See #46 for some initial
context.

Basic design is:
1. A decorator that loads data and injects into a function
2. A set of Data Loaders that can be constructed from kwargs (they're
   all dataclasses)
3. A LoadFromDecorator that takes in a variety of data loader classes
   that are possible to choose from
4. A registry that specifies a mapping of key -> data loader classes,
   allowing us to polymorphically select them

More todos, up next
1. Add tests for the default loaders
2. Add safeguards to ensure any loader is a dataclass
3. Add documentation

We also add a register_adapter function in the registry. Note that this
just moves the adapters we currently have around, and does not add new
ones.
@zilto
Copy link
Collaborator

zilto commented Jul 25, 2023

duckdb now supports Python UDF, with easy integration for functions that are typed and are pure (ref)

@skrawcz
Copy link
Collaborator

skrawcz commented Mar 26, 2024

I think we have done a form of this.

@skrawcz skrawcz closed this as completed Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
migrated-from-old-repo Migrated from old repository product idea
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@skrawcz @elijahbenizzy @zilto @HamiltonRepoMigrationBot