Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(dependencies): pandas and numpy are now optional for non-backend installs #9564

Merged
merged 21 commits into from
Aug 7, 2024
Merged

refactor(dependencies): pandas and numpy are now optional for non-backend installs #9564

merged 21 commits into from
Aug 7, 2024

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Jul 13, 2024 •
edited
Loading

Attempt to make pandas and numpy optional dependencies for installs that do not require a backend. Depends on #9552.

Closes #8430.

@cpcloud cpcloud added the dependencies Issues or PRs related to dependencies label Jul 13, 2024
@cpcloud cpcloud changed the title optional numpy and pandas refactor(dependencies): optional numpy and pandas Jul 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch 7 times, most recently from ff0ce67 to 13f24be Compare July 14, 2024 01:23
@cpcloud
Copy link
Member Author

cpcloud commented Jul 14, 2024

cc @jonmmease

After this PR, you should be able to get an Ibis installation with none of the heavier dependencies, resulting in an install size of less than 30 MB.

@jonmmease
Copy link

Very cool! What features will work with the minimal dependencies with this PR?

Will it be possible to perform sql generation for dialects (like BigQuery) without installing the execution dependencies?

@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch from aa67c58 to d9f4479 Compare July 14, 2024 15:58
@cpcloud
Copy link
Member Author

cpcloud commented Jul 14, 2024

Will it be possible to perform sql generation for dialects (like BigQuery) without installing > the execution dependencies?

Not yet :)

But that's the next step, just need to move the compilers out of the individual backends.

@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch 2 times, most recently from 2108d85 to ed72a0f Compare July 15, 2024 13:54
@cpcloud cpcloud marked this pull request as ready for review July 15, 2024 16:22
@cpcloud cpcloud mentioned this pull request Jul 16, 2024
Merged
gforsyth pushed a commit that referenced this pull request Jul 16, 2024
@cpcloud
fix(pyspark): set lower bound of pyspark to 3.3.3 to avoid maintenanc…
97af53c 
…e burden of pytest collection hook (#9606)

Increase the lower bound of pyspark to avoid a big hack in test
collection. Also reduces hacking around numpy imports when making those
optional in #9564.
@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch 3 times, most recently from 1e3f35d to 81d4d14 Compare July 16, 2024 14:50
@MarcoGorelli MarcoGorelli mentioned this pull request Jul 18, 2024
Closed
1 task
@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch from 81d4d14 to 7aaab09 Compare July 23, 2024 12:22
@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch 2 times, most recently from ae4ecb7 to 7d0c00a Compare August 2, 2024 15:18
@cpcloud
Copy link
Member Author

cpcloud commented Aug 2, 2024

Is anyone opposed to this?

To be clear, if you install a backend you still get pyarrow, numpy, and pandas.

This PR makes it possible for pip install ibis-framework without extras to be super light, so that people can use just the expression system and compilers.

@cpcloud cpcloud requested review from gforsyth and jcrist August 2, 2024 16:08
@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch from 7d0c00a to 5029b22 Compare August 4, 2024 13:01
jcrist
jcrist approved these changes Aug 7, 2024
View reviewed changes
ibis/backends/sqlite/__init__.py Outdated Show resolved Hide resolved
cpcloud added 21 commits August 7, 2024 12:15
@cpcloud
chore(deps): make numpy and pandas optional in pyproject.toml
6f5ad53
@cpcloud
chore: make pandas/numpy-less nix build for easier verification of sm…
0f03e4d 
…all dependency set
@cpcloud
ci: install pandas/numpy explicitly now that they are optional
be12eed
@cpcloud
chore(deps): move numpy and pandas requirement out of backend tests t…
4270175 
…o allow collection to succeed
@cpcloud
chore(deps): move fixtures that depend on pandas to use site
8ac750d
@cpcloud
chore(deps): allow test collection of formats when deps are not insta…
22fcfca 
…lled
@cpcloud
test: move pandas and numpy out of critical test paths
ba60bbb
@cpcloud
test: remove useless repr assertion
3848a9b
@cpcloud
test: skip sql parsing tests if black is not installed
9c22c7a
@cpcloud
chore(postgres): remove numpy and pandas entirely
afd3f65
@cpcloud
chore(oracle): remove numpy and pandas entirely
d652269
@cpcloud
chore(polars): remove numpy and pandas entirely
696215d
@cpcloud
chore(mysql): remove numpy and pandas entirely
0dc1963
@cpcloud
chore(sqlite): move converter out of the top level import path
a25c280
@cpcloud
chore(deps): move rich out of top-level import path
9716e44
@cpcloud
chore: move pandas import out pivot_wider
1cf8d6d
@cpcloud
chore(duckdb): remove numpy from converter
5c870fd
@cpcloud
chore(vectorized-udf): move numpy out of top-level import path
88930cd
@cpcloud
chore(bigquery): use builtin date and datetime types instead of relyi…
eca0cc1 
…ng on pandas
@cpcloud
fix(numpy): handle numpy timedeltas in normalize_timedelta
581842d
@cpcloud
chore: move sqlite converter up
6c6a748
@cpcloud cpcloud force-pushed the optional-numpy-and-pandas branch from 5fa2705 to 6c6a748 Compare August 7, 2024 16:17
@cpcloud cpcloud added the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Aug 7, 2024
@ibis-docs-bot ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Aug 7, 2024
@cpcloud cpcloud changed the title refactor(dependencies): optional numpy and pandas refactor(dependencies): pandas and numpy are now optional for non-backend installs Aug 7, 2024
@cpcloud cpcloud merged commit cff210a into ibis-project:main Aug 7, 2024
89 checks passed
@cpcloud cpcloud deleted the optional-numpy-and-pandas branch August 7, 2024 16:43
@lostmygithubaccount lostmygithubaccount mentioned this pull request Sep 5, 2024
Closed
@jitingxu1 jitingxu1 mentioned this pull request Sep 5, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gforsyth gforsyth gforsyth approved these changes

@kszucs kszucs kszucs left review comments

@jcrist jcrist jcrist approved these changes

Assignees
No one assigned
Labels
dependencies Issues or PRs related to dependencies
Projects
None yet
Milestone
9.3
Development

Successfully merging this pull request may close these issues.

feat: make pandas/pyarrow/numpy optional dependencies
5 participants
@cpcloud @jonmmease @gforsyth @kszucs @jcrist