Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add exact match deduplicator based on ray and redis #263

Merged
merged 15 commits into from
Mar 19, 2024
Merged

Add exact match deduplicator based on ray and redis #263

merged 15 commits into from
Mar 19, 2024

Conversation

pan-x-c
Copy link
Collaborator

@pan-x-c pan-x-c commented Mar 18, 2024

Add a simple deduplicator using Ray and Redis.
Although the purpose of this operator is deduplication, its implementation is a Filter subclass

@pan-x-c pan-x-c added enhancement New feature or request dj:op issues/PRs about some specific OPs dj:dist issues/PRs about distributed data processing labels Mar 18, 2024
@pan-x-c pan-x-c added this to the Distributed processing milestone Mar 18, 2024
@pan-x-c pan-x-c self-assigned this Mar 18, 2024
@garyzhang99 garyzhang99 changed the title [WIP] Add exact match deduplicator based on ray and redis Add exact match deduplicator based on ray and redis Mar 18, 2024
HYLcool
HYLcool reviewed Mar 18, 2024
View reviewed changes
configs/config_all.yaml Outdated Show resolved Hide resolved
yxdyc
yxdyc reviewed Mar 18, 2024
View reviewed changes
Copy link
Collaborator

@yxdyc yxdyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在operators的中英文文档也相应加一下吧，多了仨算子

HYLcool
HYLcool approved these changes Mar 19, 2024
View reviewed changes
Copy link
Collaborator

@HYLcool HYLcool left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pan-x-c pan-x-c merged commit b8e3a1b into main Mar 19, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yxdyc yxdyc yxdyc left review comments

@HYLcool HYLcool HYLcool approved these changes

Assignees

@pan-x-c pan-x-c

@garyzhang99 garyzhang99

Labels
dj:dist issues/PRs about distributed data processing dj:op issues/PRs about some specific OPs enhancement New feature or request
Projects
None yet
Milestone
Distributed processing
Development

Successfully merging this pull request may close these issues.
4 participants
@pan-x-c @HYLcool @yxdyc @garyzhang99