Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark version matrix #159

Closed
3 tasks done
alfonsorr opened this issue Jan 28, 2022 · 3 comments · Fixed by #184, #185, #187 or #186
Closed
3 tasks done

Spark version matrix #159

alfonsorr opened this issue Jan 28, 2022 · 3 comments · Fixed by #184, #185, #187 or #186
Assignees
@eruizalo @alfonsorr
Labels
CI/CD Continuous integration and continuous delivery dependencies Pull requests that update a dependency file enhancement New feature or request

Comments

@alfonsorr
Copy link
Member

alfonsorr commented Jan 28, 2022 •
edited by eruizalo
Loading

Review how to publish doric according to the spark version.
Check dependencies for all versions.
@alfonsorr alfonsorr added enhancement New feature or request dependencies Pull requests that update a dependency file CI/CD Continuous integration and continuous delivery labels Jan 28, 2022
@alfonsorr
Copy link
Member Author

Sadly spark 2.4 won't be built against due to problems with dependencies in scala 2.11

@alfonsorr alfonsorr self-assigned this Feb 25, 2022
@alfonsorr alfonsorr mentioned this issue Mar 1, 2022
Merged
1 task
@eruizalo eruizalo linked a pull request Mar 2, 2022 that will close this issue
Cross build doric from Spark 3.0 to 3.2 #184
Merged
1 task
eruizalo pushed a commit that referenced this issue Mar 3, 2022
@alfonsorr
Cross build doric from Spark 3.0 to 3.2 (issue #159) (#184)
41f1766
@eruizalo eruizalo reopened this Mar 3, 2022
@alfonsorr alfonsorr mentioned this issue Mar 3, 2022
Merged
1 task
@eruizalo eruizalo linked a pull request Mar 4, 2022 that will close this issue
Single parameter sparkVersion allows 3.x and 3.x.x versions #185
Merged
1 task
eruizalo pushed a commit that referenced this issue Mar 4, 2022
@alfonsorr
Single parameter sparkVersion allows 3.x and 3.x.x versions (issue #159…
a281269 
…) (#185)
@eruizalo eruizalo reopened this Mar 4, 2022
@eruizalo eruizalo mentioned this issue Mar 6, 2022
Merged
1 task
@eruizalo eruizalo linked a pull request Mar 7, 2022 that will close this issue
ci: [+] add spark labels to labeler #186
Merged
1 task
eruizalo added a commit to eruizalo/doric that referenced this issue Mar 9, 2022
@eruizalo
ci: [+] add matrix to CI workflow (hablapps#159)
40718e3
@eruizalo eruizalo self-assigned this Mar 15, 2022
eruizalo added a commit that referenced this issue Mar 18, 2022
@eruizalo
ci/#159 spark version matrix (issue #159) (#187)
bcd73e3 
* ci: [+] add matrix to CI workflow (#159)
* fix link & codecov labels
* ci: [~] fix checkout full history
* ci: [~] update publish_doc action
* ci: [–] delete unnecessary clean cache steps
@eruizalo eruizalo reopened this Mar 18, 2022
@nightscape
Copy link

Hi @alfonsorr, I'm working in a project where we're tied to Scala 2.11 and Spark 2.4.7 (Cloudera 😵‍💫).
Doric could be a great match for a use-case and I might get some time to invest in backporting.
Do you still remember what the dependency problems you mentioned were and where you got stuck?
I have some experience with Spark plugin building and packaging (I'm one of the maintainers of https://github.com/crealytics/spark-excel/) so I might be able to pick this up.

@alfonsorr
Copy link
Member Author

Hi @nightscape . You are one of the first postings in our repo. We are so happy 😄 Any help is welcome.
Yeah, we wanted to do a fast cross spark version of doric and didn't want to change first a lot of code, we wanted to migrate also to 2.4 but required a little more work than expected.

There are 3 dependencies that give some problems, and I suggest that what we should do with them are:

  • Newtype doesn't have a version for 2.11, but we can get rid of it and change it to a case class where it is used.
  • spark daria we can check if the versions published for 2.11 have all the logic we need, if not we should replace with something custom-made.
  • cats was the main reason that I didn't make in this first iteration of the porting to spark 2.4, I've changed the dependency to 2.0.0 (last 2.11 published) and started to pop a lot of import errors and didn't want to change it so much. But I think that for what we are using it should be achievable.

We are currently trying to release the 0.0.3 this week that will be the one with cross-version from 3.0 to 3.2, but then we can check it with more time to try to port it to 2.4 at least. For the moment the changes are only in the CI files.

Feel free to create an issue if you start to do any code and we will get into also when we finish the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eruizalo eruizalo

@alfonsorr alfonsorr

Labels
CI/CD Continuous integration and continuous delivery dependencies Pull requests that update a dependency file enhancement New feature or request
Projects
Habla doric
Status: Done
Milestone
No milestone
3 participants
@nightscape @eruizalo @alfonsorr