Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow to install PIP packages into PySpark job #215

Merged
merged 19 commits into from
Dec 14, 2023
Merged

Allow to install PIP packages into PySpark job #215

merged 19 commits into from
Dec 14, 2023

Conversation

EnricoMi
Copy link
Contributor

No description provided.

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2023 •
edited
Loading

Test Results

     566 files  ±    0       566 suites  ±0   1h 32m 19s ⏱️ + 2m 53s
     547 tests +  11       547 ✔️ +  11    0 💤 ±  0  0 ❌ ±0 
17 004 runs  +176  16 970 ✔️ +144  34 💤 +32  0 ❌ ±0 

Results for commit 8ed799b. ± Comparison against base commit ac6f6cb.

♻️ This comment has been updated with latest results.

@EnricoMi
Add install_pip_dependencies
c3bf736
@EnricoMi
Make install method more rebust
4d1daf1
@EnricoMi
Add create dir method to readme, add tests, add to Python
e266a84
@EnricoMi
Fix temp dir
14cbe98
@EnricoMi
Add pyarrow to test dependencies
ff5f5d8
@EnricoMi
Add pandas to python test dependencies
7b23583
@EnricoMi
Support pip install only from PySpark 3.1
d9fe9df
@EnricoMi
Test with mapInPandas, not mapInArrow
caa3b78
@EnricoMi
Add pip install to README.md, add more details sections
259f9c8
@EnricoMi
PyArrow still needed
dfd12f1
@EnricoMi
Handle pip failures gracefully
bc9b42e
@EnricoMi
Add dedicated MD for installing python packages, use more details sec…
8d83ef4 
…tions
@EnricoMi
Run pip via subprocess
58d8b45
@EnricoMi
Rename to install_pip_package
f4dcf55
@EnricoMi
More examples
81869dc
@EnricoMi
Fix method name in PYSPARK-DEPS.md
0dd75c1
@EnricoMi
Fix import in README.md
9760c44
@EnricoMi
Revert changes to pom.xml and setup.py
801e607
@EnricoMi
Run pip through subprocess.check_call
8ed799b
@EnricoMi EnricoMi merged commit 57aec01 into master Dec 14, 2023
87 checks passed
@EnricoMi EnricoMi deleted the pip_install branch December 14, 2023 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@EnricoMi