Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Add spark connector #1809

Merged
merged 4 commits into from
Jul 11, 2023
Merged

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport a816a58 from #1780

* Create Spark Connector

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

* Add spark client and engine

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Remove vars

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Spark connector draft

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* nit

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix checkstyle errors

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* nit

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix license header

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add spark storage test

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update comments

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix checkstyle in comments

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update tests

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Address PR comments

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Refactor class name

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Address PR comment

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

---------

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>
(cherry picked from commit a816a58)
@codecov
Copy link

codecov bot commented Jul 5, 2023

Codecov Report

Merging #1809 (bbe929d) into 2.x (776a76c) will increase coverage by 0.01%.
The diff coverage is 100.00%.

❗ Current head bbe929d differs from pull request most recent head c4aebdd. Consider uploading reports for the commit c4aebdd to get more accurate results

@@             Coverage Diff              @@
##                2.x    #1809      +/-   ##
============================================
+ Coverage     97.35%   97.37%   +0.01%     
- Complexity     4443     4475      +32     
============================================
  Files           388      394       +6     
  Lines         10997    11065      +68     
  Branches        784      789       +5     
============================================
+ Hits          10706    10774      +68     
  Misses          284      284              
  Partials          7        7              
Flag Coverage Δ
sql-engine 97.37% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...implementation/SparkSqlFunctionImplementation.java 100.00% <100.00%> (ø)
...ctions/resolver/SparkSqlTableFunctionResolver.java 100.00% <100.00%> (ø)
...nctions/scan/SparkSqlFunctionTableScanBuilder.java 100.00% <100.00%> (ø)
...ensearch/sql/spark/storage/SparkStorageEngine.java 100.00% <100.00%> (ø)
...nsearch/sql/spark/storage/SparkStorageFactory.java 100.00% <100.00%> (ø)
...a/org/opensearch/sql/spark/storage/SparkTable.java 100.00% <100.00%> (ø)

Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@penghuo penghuo merged commit a3d9bfc into 2.x Jul 11, 2023
19 of 20 checks passed
@github-actions github-actions bot deleted the backport/backport-1780-to-2.x branch July 11, 2023 16:24
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants