fix: prase edge error when data props num is different #211
Conversation
Codecov Report
@@ Coverage Diff @@
## master #211 +/- ##
============================================
- Coverage 74.04% 74.03% -0.02%
Complexity 871 871
============================================
Files 80 80
Lines 3645 3647 +2
Branches 436 437 +1
============================================
+ Hits 2699 2700 +1
Misses 745 745
- Partials 201 202 +1
Continue to review full report at Codecov.
|
imbajin
changed the title
| if (this.vertexIdsIndex == null) { | ||
| this.vertexIdsIndex = this.extractVertexIdsIndex(names); | ||
| } | ||
| this.vertexIdsIndex = this.extractVertexIdsIndex(names); |
There was a problem hiding this comment.
每次build都重新提取下标开销太大。
图中这个提交将属性的获取从map变成了数组下标访问的方式，你碰到的问题应该是由这个提交导致的。
请切到这个提交的前面一个提交，验证一下是否存在你的那个问题。如果不存在了，我觉得还是得改回用map访问的方式。
There was a problem hiding this comment.
改回map的话 ，又回到了每行json创建map的情况。
如果对names进行判断，names代表json数据的key值，他没有变化就说明这行数据的索引位置和数值都没变，那么就只有在json的key变化的时候重新计算就可以了
There was a problem hiding this comment.
文件是Json的时候，现有代码是一定会生成一次map的，性能无影响的；受影响的是文件是CSV或TEXT格式的时候。
所以现在的方案就是两个：
- 所有文件格式都先生成map，无需提取数组索引，这样CSV、TEXT有性能损耗，JSON无影响；
- CSV、TEXT直接生成数组，JSON先生成map再转成数组，然后CSV、TEXT无性能损耗，JSON需要对比和重新提取，有损耗。
综合考虑，建议采用方案2，可做这样修改：
- 文件是CSV和TEXT的时候，names其实每次都是header引用的那个对象，所以不会有变化，不用重新提取索引；每次都比较（或者用==比较的方式）；
- 文件是JSON格式时，先对比，如果有改变就重新提取。
代码大概是：
private String[] lastNames;
...
public List<Edge> build(String[] names, Object[] values) {
if (this.vertexIdsIndex == null ||
(this.lastNames != names && !Arrays.equals(this.lastNames, names))) {
this.vertexIdsIndex = this.extractVertexIdsIndex(names);
}
this.lastNames = names;
...
}
| if (this.vertexIdsIndex == null) { | ||
| this.vertexIdsIndex = this.extractVertexIdsIndex(names); | ||
| } | ||
| this.vertexIdsIndex = this.extractVertexIdsIndex(names); |
DoubleBabylol
force-pushed
the
jsonlaodfix
branch
from
e313e4f to
e479ba4
Compare
| import java.util.ArrayList; | ||
| import java.util.Collection; | ||
| import java.util.HashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.Set; | ||
| import java.util.*; |
There was a problem hiding this comment.
keep original, use this config file with IDEA
src/main/java/com/baidu/hugegraph/loader/builder/EdgeBuilder.java
Outdated
Show resolved
Hide resolved
DoubleBabylol
force-pushed
the
jsonlaodfix
branch
from
36a651a to
5f9aa9b
Compare
| @@ -69,9 +71,12 @@ public EdgeMapping mapping() { | |||
|
|
|||
| @Override | |||
| public List<Edge> build(String[] names, Object[] values) { | |||
| if (this.vertexIdsIndex == null) { | |||
| if (this.vertexIdsIndex == null || | |||
| (this.lastNames != names && | |||
There was a problem hiding this comment.
no need to compare address because Arrays.equals() will compare them first
fix #209