-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT-#1838: Lazy map evaluation at Pandas backend #1940
Conversation
fc314af
to
5e62ad9
Compare
Codecov Report
@@ Coverage Diff @@
## master #1940 +/- ##
==========================================
+ Coverage 76.63% 81.76% +5.13%
==========================================
Files 79 79
Lines 9303 9303
==========================================
+ Hits 7129 7607 +478
+ Misses 2174 1696 -478
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you replace part.apply
/obj.apply
to part.add_to_apply_calls
/obj.add_to_apply_calls
in broadcast_apply
, _apply_func_to_list_of_partitions
and _apply_func_to_list_of_partitions_broadcast
only. We have many places where part.apply
/obj.apply
is used. Can we replace this somewhere else?
there are a few places where we still have
|
|
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
@YarShev replacing Call queue is the partitions abstraction, and it must be explicitly drained with a special partition method before every serialization. So if we'll do So, there are two different solutions:
new_idx = [part.add_to_apply_calls(fn) for part in partitions]
new_idx = [part.drain_call_queue() for part in partitions]
new_idx = [part.oid for part in partitions]
return ray.get(new_idx) Which is looks kinda strange, and do not have any real advantages before just doing |
@dchigarev , okay, I see. Let's keep |
What do these changes do?
flake8 modin
black --check modin
git commit -s
MapFunction
is very slow at Ray engine #1838