-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Spark loader Task not serializable #471
Conversation
[Bug] Spark loader Task not serializable
@@ -77,7 +77,7 @@ public class HugeGraphSparkLoader implements Serializable { | |||
private final LoadOptions loadOptions; | |||
private final Map<ElementBuilder, List<GraphElement>> builders; | |||
|
|||
private final ExecutorService executor; | |||
private final transient ExecutorService executor; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it the root problem? And how to ensure it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, i think the root cause of the problem is that we later used the specific properties of hugegraphsparkloader when building the vertex and edge data, such as: loadoptions,builders; spark needs to transfer the entire hugegraphsparkloader object over the network, so it requires both the hugegraphsparkloader itself and its properties to implement a serialization interface. However, the ExecutorService type does not implement serialization, so it throws an error that the task cannot be serialized. Using transient allows us to avoid serializing the ExecutorService executor when serializing hugegraphsparkloader, because we will not need the ExecutorService executor during the vertexs/edges construction process after submitting the job. So I think we can solve this problem by adding transient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make sense
Codecov Report
@@ Coverage Diff @@
## master #471 +/- ##
=========================================
Coverage 62.51% 62.51%
Complexity 894 894
=========================================
Files 91 91
Lines 4396 4396
Branches 516 516
=========================================
Hits 2748 2748
Misses 1445 1445
Partials 203 203
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
fix #467