Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve RPyC connection performance #1821

Closed
vnlitvinov opened this issue Jul 27, 2020 · 0 comments · Fixed by #1833
Closed

Improve RPyC connection performance #1821

vnlitvinov opened this issue Jul 27, 2020 · 0 comments · Fixed by #1833
Assignees
@vnlitvinov
Labels
new feature/request 💬 Requests and pull requests for new features
Milestone
0.8.0

Comments

@vnlitvinov
Copy link
Collaborator

Currently (as of #1818) RPyC is being used as the main communication channel between local Modin and scaled-out to the cluster.
It does a lot of excessive calls, and each call costs at least as much time as ping to spawned cluster (because most of the calls need to be synchronous).

We need to make less calls and cache whatever is safe.
@vnlitvinov vnlitvinov added the new feature/request 💬 Requests and pull requests for new features label Jul 27, 2020
@vnlitvinov vnlitvinov added this to the 0.8.0 milestone Jul 27, 2020
@vnlitvinov vnlitvinov self-assigned this Jul 27, 2020
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Implement some caching
54e310e 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Cache remote class types from numpy and pand…
043507c 
…as, too

Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Cache classes for pandas.* and numpy.*
35f06f7 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Try making a faster connection
3008671 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Fix support for proxying methods RPyC claims…
ffd3113 
… are local

Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: More efficient, batching delivering
864b267 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Some improvements in Cloudray factory and stuff
7087cec 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Add numpy attrs caching
e8027e2 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Add obtaining of df.dtypes.items()
82709fe 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Fix cache going stale
c04aa34 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Allow subclassed mixins
996e6c8 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Define .items() simpler
0e99678 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Add caching of numpy things hashes
a1b75fb 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Make caching more correct
fdeb26e 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Format stuff by black
5a2c2b5 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
@vnlitvinov vnlitvinov mentioned this issue Jul 28, 2020
Merged
6 tasks
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Add missing import (minor)
c1c86f5 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
vnlitvinov added a commit to vnlitvinov/modin that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-modin-project#1821: Add comments and docstrings
99650d7 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
devin-petersohn pushed a commit that referenced this issue Jul 28, 2020
@vnlitvinov
FEAT-#1821: Speed up RPyC connection (#1833)
0a03e7a 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
aregm pushed a commit to aregm/modin that referenced this issue Sep 16, 2020
@vnlitvinov @aregm
FEAT-modin-project#1821: Speed up RPyC connection (modin-project#1833)
276daa0 
Signed-off-by: Vasilij Litvinov <vasilij.n.litvinov@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vnlitvinov vnlitvinov

Labels
new feature/request 💬 Requests and pull requests for new features
Projects
None yet
Milestone
0.8.0
Development

Successfully merging a pull request may close this issue.

Speed up RPyC connection vnlitvinov/modin
1 participant
@vnlitvinov