Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support incremental loading data for 'vineyard-graph-loader' #1563

Merged
merged 2 commits into from
Sep 26, 2023

Conversation

SighingSnow
Copy link
Contributor

@SighingSnow SighingSnow commented Sep 12, 2023

What do these changes do?

Add support for incremental vertices and edges for vineyard-graph-loader.
For step_by_step, the expected behaviors are as below

Order Case Expect Behavior
1 combine multi files of same label as before vineyard-graph-loader
2 combine multi files of same label but with duplicated data(vertex) drop duplicates
3 combine multi files of same label but with duplicated data(edge) log warning and becom multigraph(as previous behaviors)

For whole or the default way, the expected behaviors are as below,

Order Case Expect Behavior
1 combine multi files of same label as before vineyard-graph-loader
2 combine multi files of same label but with duplicated data(vertex) expect user not to add this kind of data
3 combine multi files of same label but with duplicated data(edge) expect user not to add this kind of data

Related issue number

FIXES #1295

@siyuan0322 would you mind helping me review of the code

@siyuan0322 siyuan0322 self-requested a review September 12, 2023 07:26
@netlify
Copy link

netlify bot commented Sep 12, 2023

Deploy Preview for v6d ready!

Name Link
🔨 Latest commit e6491ef9af6cdb59040a28607212a0e9870a01d9
🔍 Latest deploy log https://app.netlify.com/sites/v6d/deploys/6502c2f5b0af670007741755
😎 Deploy Preview https://deploy-preview-1563--v6d.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@SighingSnow SighingSnow force-pushed the sty/progress branch 2 times, most recently from e103c02 to bc6471d Compare September 12, 2023 13:24
Copy link
Member

@siyuan0322 siyuan0322 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your efforts for addressed this tough task!
Overall looks good, and some implementation details need to be discussed.

.cirrus.yml Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
modules/graph/fragment/arrow_fragment_builder_impl.h Outdated Show resolved Hide resolved
modules/graph/fragment/arrow_fragment_builder_impl.h Outdated Show resolved Hide resolved
modules/graph/loader/basic_ev_fragment_loader.h Outdated Show resolved Hide resolved
modules/graph/loader/basic_ev_fragment_loader_impl.h Outdated Show resolved Hide resolved
modules/graph/tools/fragment_loader_impl.h Show resolved Hide resolved
modules/graph/tools/fragment_loader_impl.h Show resolved Hide resolved
modules/graph/vertex_map/arrow_vertex_map_impl.h Outdated Show resolved Hide resolved
@siyuan0322
Copy link
Member

And remember to use clang-format to format your codes before pushing, you could find the way to use it in the CI yamls.

@SighingSnow SighingSnow force-pushed the sty/progress branch 2 times, most recently from f47c218 to 0acbcb4 Compare September 21, 2023 03:51
@@ -13,6 +13,7 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

#include <_types/_uint32_t.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#include <_types/_uint32_t.h>

@@ -13,6 +13,7 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

#include <_types/_uint64_t.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not delete these headers?

Copy link
Member

@sighingnow sighingnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @SighingSnow could you please revert the etcd-cpp-apiv3 submodule changes?

Signed-off-by: SighingSnow <1263750383@qq.com>
Signed-off-by: SighingSnow <1263750383@qq.com>
@SighingSnow SighingSnow force-pushed the sty/progress branch 3 times, most recently from 05821c5 to 7559a79 Compare September 26, 2023 03:19
@siyuan0322 siyuan0322 merged commit 1de8226 into v6d-io:main Sep 26, 2023
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support adding the same label edges with different labels in src and dst
3 participants