Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The write-generated-reference-tables shouldn't overwrite tables by default #11

Open
MrPowers opened this issue Oct 10, 2022 · 1 comment
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@MrPowers
Copy link
Collaborator

When I run write-generated-reference-tables, I'll normally only want to add the new reference tables that are missing. I won't want to overwrite the existing reference tables. Would be cool to not overwrite by default and add a flag that needs to be set if you want them to all be overwritten.

@MrPowers MrPowers added enhancement New feature or request good first issue Good for newcomers labels Oct 10, 2022
@edmondop
Copy link
Contributor

One fundamental challenge here is that maybe some code to generate the table has been modified, and therefore you really want to regenerate existing tables.

We can leave the option to the user to decide, or write the parquet data to a tmp folder and then read the two tables and compare to make the process idempotent, or also publish in a json file more information about the data which will be generated (but it only works for small data)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Development

No branches or pull requests

2 participants