Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce HugeGraphSparkLoader #282

Merged
merged 9 commits into from
May 27, 2022
Merged

feat: introduce HugeGraphSparkLoader #282

merged 9 commits into from
May 27, 2022

Conversation

simon824
Copy link
Member

@simon824 simon824 commented May 24, 2022

closed #281
basic version of HugeGraphSparkLoader

test cmd
The options are divided into two parts, spark options and hugegraph-loader options. (no sorting required)

 ./bin/hugegraph-spark-loader.sh  \
--master yarn --deploy-mode client --name spark-test \
--conf spark.eventLog.enabled=false \
--conf spark.executor.extraJavaOptions=-XX:+PrintGCDetails\
-f ./conf/spark.json --username admin --token admin \
-h 127.0.0.1 -p 8093 -g my_graph2 

@codecov
Copy link

codecov bot commented May 24, 2022

Codecov Report

Merging #282 (ba7dedf) into master (7c39453) will decrease coverage by 4.61%.
The diff coverage is 1.72%.

@@             Coverage Diff              @@
##             master     #282      +/-   ##
============================================
- Coverage     75.74%   71.12%   -4.62%     
+ Complexity     1847      877     -970     
============================================
  Files           235       82     -153     
  Lines          7766     3816    -3950     
  Branches        685      456     -229     
============================================
- Hits           5882     2714    -3168     
+ Misses         1501      898     -603     
+ Partials        383      204     -179     
Impacted Files Coverage Δ
...m/baidu/hugegraph/loader/executor/LoadContext.java 67.90% <0.00%> (ø)
...om/baidu/hugegraph/loader/mapping/InputStruct.java 93.02% <ø> (ø)
.../baidu/hugegraph/loader/source/AbstractSource.java 83.33% <ø> (ø)
...baidu/hugegraph/loader/source/file/FileFilter.java 75.00% <ø> (ø)
...aidu/hugegraph/loader/source/file/SkippedLine.java 91.66% <ø> (ø)
...u/hugegraph/loader/spark/HugeGraphSparkLoader.java 0.00% <0.00%> (ø)
...baidu/hugegraph/loader/mapping/ElementMapping.java 87.83% <50.00%> (-1.06%) ⬇️
...m/baidu/hugegraph/loader/executor/LoadOptions.java 74.41% <100.00%> (ø)
.../hugegraph/structure/graph/BatchVertexRequest.java
.../baidu/hugegraph/structure/schema/SchemaLabel.java
... and 152 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c39453...ba7dedf. Read the comment docs.

Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, it's a very nice enhancement. Some tiny comments

@imbajin
Copy link
Member

imbajin commented May 24, 2022

could enable check-style plugin to check code

javeme
javeme previously approved these changes May 24, 2022
Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imbajin imbajin merged commit 9ee7abb into apache:master May 27, 2022
@javeme javeme added the loader hugegraph-loader label Jul 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
loader hugegraph-loader
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Introduce HugeGraphSparkLoader
3 participants