Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chunks externalized as Blobs.jl #23

Merged
merged 5 commits into from
Mar 16, 2016
Merged

chunks externalized as Blobs.jl #23

merged 5 commits into from
Mar 16, 2016

Conversation

tanmaykm
Copy link
Collaborator

@tanmaykm tanmaykm commented Mar 9, 2016

Chunks, introduced in #22 are now externalized to a separate package https://github.com/tanmaykm/Blobs.jl

@tanmaykm
Copy link
Collaborator Author

Travis tests fail because Blobs is not registered yet.

Performance figures with recent optimizations, under the same conditions as in #22:

machine dataset mode time (sec)
laptop 4D chunks/blobs 1113
laptop D chunks/blobs 414
laptop D shmem (@parallel) 286
laptop D threads 525

Threads mode is probably slower because of gc?

@tanmaykm
print time with logs, some optimizations
c3a3a04 
- club together remotecalls as much as possible
- use sync-async structure to share data prior to factorization loops
- remove skew from lastfm
abhijithch added a commit that referenced this pull request Mar 16, 2016
@abhijithch
Merge pull request #23 from tanmaykm/distributed
84813ad 
chunks externalized as Blobs.jl
@abhijithch abhijithch merged commit 84813ad into abhijithch:master Mar 16, 2016
@tanmaykm
Copy link
Collaborator Author

Whoops! I should have mentioned this was still WIP.

@tanmaykm
Copy link
Collaborator Author

The PR was in an inconsistent state with some recent changes I have been doing for optimization. Should probably undo the merge till the following are finished:

  • verification of rmse, they are higher than expected
  • optimization of rmse calculation, I think they can be vectorized. It is otherwise painful to verify calculations of large sized inputs.
  • use memory limits with blobs instead of blob count limit. This will avoid need to change code/parameters to run on differently sized machines.

@abhijithch abhijithch mentioned this pull request Mar 16, 2016
Merged
@tanmaykm tanmaykm mentioned this pull request Mar 22, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tanmaykm @abhijithch