Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Allow user provided schema and schema inference length for read_sql #2676

Merged
merged 2 commits into from
Aug 20, 2024

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Aug 16, 2024

Enables users to provide a schema to read_sql, as well as an infer_schema flag to turn on/off inference, and infer_schema_length to customize the length of the scan for inference.

Note that infer_schema_length only applies when SQLAlchemy is used to read, otherwise ConnectorX is able to provide the schema with a limit=0 as it is an arrow backed driver. This is a new optimization.

This PR also has a few refactors, namely moving the construct_sql_query and the read_schema logic to the SQLConnection object instead of the ScanOperator. This is so that schema inference also has context on the driver that will be used.

@github-actions github-actions bot added the enhancement New feature or request label Aug 16, 2024
@colin-ho colin-ho requested review from jaychia and samster25 and removed request for jaychia August 19, 2024 17:57
@samster25 samster25 merged commit f34837d into main Aug 20, 2024
50 checks passed
@samster25 samster25 deleted the colin/sql-schema-inference branch August 20, 2024 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants