Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update spark submit operator for spark 3 support #8730

Merged

Conversation

stijndehaes
Copy link
Contributor

@stijndehaes stijndehaes commented May 6, 2020

In spark 3 they log the exit code with a lowercase
e, in spark 2 they used an uppercase E.

Also made the exception a bit clearer when running
on kubernetes.


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

Copy link

@dawany dawany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compatible with spark2 and 3

@stale
Copy link

stale bot commented Jul 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 11, 2020
@stijndehaes
Copy link
Contributor Author

This issue is still very relevant more so with the official release of spark 3.0

@stale stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 13, 2020
@stijndehaes stijndehaes force-pushed the feature/spark-submit-operator-spark-3 branch from d01fc7d to 6265060 Compare July 13, 2020 12:03
In spark 3 they log the exit code with a lowercase
e, in spark 2 they used an uppercase E.

Also made the exception a bit clearer when running
on kubernetes.
@stijndehaes stijndehaes force-pushed the feature/spark-submit-operator-spark-3 branch from 6265060 to 7243edf Compare July 22, 2020 04:53
@stijndehaes stijndehaes requested a review from kaxil July 22, 2020 04:54
@kaxil kaxil merged commit 1427e4a into apache:master Jul 22, 2020
@kaxil
Copy link
Member

kaxil commented Jul 22, 2020

Thanks for the fix @stijndehaes

@teddyhartanto
Copy link
Contributor

Thank you! @stijndehaes @kaxil

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants