Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distributed Runtime #218

Merged
196 commits merged into from
Nov 29, 2022
Merged

Distributed Runtime #218

196 commits merged into from
Nov 29, 2022

Conversation

ryanolson
Copy link
Contributor

@ryanolson ryanolson commented Nov 21, 2022 •
edited by mdemoret-nv
Loading

This PR is a major step toward the completion Multi-Node or fully distributed SRF pipelines.

This PR puts in place the framework for the following components:

  • control plane
  • data plane
  • fiber based grpc
  • ucx runtime

This PR improves upon the following existing components:

  • codable protocol

This PR add the initial public runtime and resources interface.

Closes #120

ryanolson added 30 commits June 16, 2022 21:11
@ryanolson
remove shared_ptr creep in system and resources
8ab6473
@ryanolson
re-enable test_runnable
e740aa2
@ryanolson
re-enable internal tests
780be79
@ryanolson
adding system::IResource to enable external thread initializers/final…
358b992 
…izers
@ryanolson
update python to create a system::IResource to initializer the executor
15ee92c
@ryanolson
iwyu
559000f
@ryanolson
removed old code
8014e6b
@ryanolson
merge main
9bec540
@ryanolson
iwyu
4ffc80e
@ryanolson
checkpoint
e05fd8d
@ryanolson
adding missing copyright header
dad1387
@ryanolson
adding network resources and re-enabling data_plane::Server
58c1649
@ryanolson
adding cpp files to build
a26bb6e
@ryanolson
fixing missing class method that results in unresovled external; clea…
5f3ea5b 
…n up headers
@ryanolson
merge updates from resource_management
a0f7c68
@ryanolson
checkpoint
2db7fba
@ryanolson
partition resources
a7cde72
@ryanolson
starting tests for network
350f5b1
@ryanolson
checkpoint
ec957da
@ryanolson
checkpoint
3a63933
@ryanolson
checkpoint
5615a03
@ryanolson
logging and arena resources seem to be working; resources::Manager bu…
dcc1900 
…ilding runnable, ucx and host memory resources
@ryanolson
increase log levels
9591532
@ryanolson
merge branch-22.08
e8363c5
@ryanolson
missing files
56c584a
@ryanolson
deleted old files left from merge
6503b11
@ryanolson
removed file
4a7b1fd
@ryanolson
moving test_memory to internal tests; adding direct registration reso…
e819f89 
…urce for device memory
@ryanolson
adding direct registration resource for device memory
291ac6f
@ryanolson
remove <cuda/memory_resource>
0ea1f51
mdemoret-nv
mdemoret-nv approved these changes Nov 22, 2022
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with comments.

cmake/dependencies.cmake Show resolved Hide resolved
include/srf/codable/api.hpp Show resolved Hide resolved
include/srf/concepts/invocable.hpp Outdated Show resolved Hide resolved
include/srf/memory/buffer_view.hpp Outdated Show resolved Hide resolved
include/srf/memory/buffer_view.hpp Outdated Show resolved Hide resolved
protos/CMakeLists.txt Show resolved Hide resolved
python/srf/_pysrf/include/pysrf/plugins.hpp Outdated Show resolved Hide resolved
src/internal/pubsub/publisher.hpp Outdated Show resolved Hide resolved
src/internal/pubsub/subscriber.hpp Outdated Show resolved Hide resolved
include/srf/pubsub/publisher.hpp Show resolved Hide resolved
ryanolson and others added 22 commits November 22, 2022 05:39
@ryanolson
renaming internal publisher/subscriber to have a service suffix
de4057d
@ryanolson
removing concepts directory which accidentially got added from a cpp2…
46bb725 
…0 branch
@ryanolson
fixed Python PluginProxy header that had some merge crud added to it
b6b64f4
@ryanolson
adding TagID to srf/types.hpp
4e76db0
@ryanolson
replacing tag_t with TagID
1a7480e
@ryanolson
removing some debugging statements
3640b93
@ryanolson
skipping pub sub tests until the control plane update is finished
70cec46
@ryanolson
refactoring the codable protocol for srf::memory::buffer; adding an i…
bdfcbbb 
…mplementation file
@ryanolson
clang-format/tidy fixes
ad35b6d
@ryanolson
iwyu
403ca93
@ryanolson
tidy
f90dc3e
@ryanolson
iwyu
9ec816d
@ryanolson
removed python codable tests until we have access to a codable object
e9d2136
@ryanolson
iwyu
8783726
@ryanolson
issue nv-morpheus#219
941f33a
@ryanolson
iwyu/tidy
0ea8c0c
@ryanolson
iwyu
2fcfefe
@ryanolson
iwyu
f591060
@ryanolson
removing nvprc
6922bcb
@drobison00
Ensure interpreter is properly initialized, minor test fixes
589d455
@ryanolson
Merge remote-tracking branch 'origin/pull-request/220' into codable_v…
b36d44c 
…3-23.01
@ryanolson
iwyu
16c150f
@mdemoret-nv mdemoret-nv mentioned this pull request Nov 28, 2022
Closed
@codecov
Copy link

codecov bot commented Nov 29, 2022

Codecov Report

Base: 32.61% // Head: 28.39% // Decreases project coverage by -4.21% ⚠️

Coverage data is based on head (24edae0) compared to base (b4c7bd7).
Patch coverage: 0.97% of modified lines in pull request are covered.

Additional details and impacted files 
@@               Coverage Diff                @@
##           branch-23.01     #218      +/-   ##
================================================
- Coverage         32.61%   28.39%   -4.22%     
================================================
  Files               322      363      +41     
  Lines              9107    10121    +1014     
  Branches           4251     4783     +532     
================================================
- Hits               2970     2874      -96     
- Misses             3810     4968    +1158     
+ Partials           2327     2279      -48
Impacted Files Coverage Δ
include/srf/codable/api.hpp 0.00% <0.00%> (ø)
include/srf/codable/decode.hpp 0.00% <0.00%> (-100.00%) ⬇️
include/srf/codable/encode.hpp 0.00% <0.00%> (-80.00%) ⬇️
include/srf/codable/encoded_object.hpp 0.00% <0.00%> (-100.00%) ⬇️
include/srf/codable/fundamental_types.hpp 0.00% <0.00%> (-46.16%) ⬇️
include/srf/codable/storage_forwarder.hpp 0.00% <0.00%> (ø)
include/srf/codable/type_traits.hpp 0.00% <0.00%> (-100.00%) ⬇️
include/srf/control_plane/api.hpp 0.00% <0.00%> (ø)
include/srf/core/task_queue.hpp 57.14% <ø> (ø)
include/srf/memory/buffer.hpp 0.00% <0.00%> (ø)
... and 132 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@mdemoret-nv
Copy link
Contributor

@gpucibot merge

@ghost ghost merged commit 9fbd4a5 into nv-morpheus:branch-23.01 Nov 29, 2022
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@ryanolson ryanolson

Labels
breaking Breaking change feature request New feature or request
Projects
No open projects
MRC Boards
Status: Done
Milestone
Multi-Node Support
Development

Successfully merging this pull request may close these issues.

[FEA] Add convenient methods to EncodedObject
3 participants
@ryanolson @mdemoret-nv @drobison00