Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The new data transfer #55

Merged
merged 24 commits into from
Jul 8, 2020
Merged

The new data transfer #55

merged 24 commits into from
Jul 8, 2020

Conversation

hannahhoward
Copy link
Collaborator

Goals

Support many things in data transfer, including:

  • pausing and resuming
  • piggy backing on graphsync for payment protocols like retrieval (via intermedia vouchers and the like)
  • actually being able to potentially support protocols other than graphsync
  • potentially support pause/resume across restarts
  • support pausing and resuming from either side

Implementation

There are a number of changes here, but the basic architecture decisions are:

  1. Factor out transport specific details from the machinery of data transfer itself -- which handles tracking tranfers, managing states of channels (including pausing and resuming, sending vouchers around). This is still a bit WIP but there's a much greater separation, which enables the non-transport specific implementation to focus on the significantly more complicated logic of allowing not simply one-and-done transfers but transfers that can stop and start, and have many intermediate points in the middle.
  2. Move channel tracking to the oft-used FSM architecture we've used for state tracking elsewhere - while perhaps we've used this hammer so much everything looks like a nail, the ability to easily specify state transition logic makes it simply to track what's happening when and have more complciated sets of steps.
  3. Actually unit test this library, then write stress test level integration tests. In the initial verison of the library, in part because separation between the transport and transfer logic was minimal, most of our tests were effectively integration tests, but at the same time they weren't testing all that much. The new version leans heavily on unit testing, but also includes integration tests that test a lot more than the old version did.

add send voucher function to data transfer, and create update options for send & receive
add utilties to channel state and extract integration tests
add a pause attribute message
Setup pull requests to immediately send a GS request
Move all transport specific actions outside of the manage, into a transport layer than abstracts the
underlying transport medium
track channel status, add new events and statuses and move start/stop out of constructor
refactor receiver to use transport to handle responses
Conver Channel to cbor-gen type
Refactor channels to use FSM, so we can track statuses and publish events automatically
also add some voucher result processing. this includes a major refactor of the testing strategy for
the implementation to be more of a unit test
Complete implementation of data transfer pause and resume
Modify voucher validation to support pauses on first validate
add the final step for a retrieval flow -- being able to stop and start via vouchers
support connection tagging to protect connections from getting dropped
Split up functions into files so that they group better
@hannahhoward hannahhoward force-pushed the feat/the-new-data-transfer branch 2 times, most recently from 39f3e94 to 7b44091 Compare July 7, 2020 15:57
switch integration test to table test to make sure not to reuse constructs taht are already in use
Only resume transport when it's possible for the transport to be resumed
Copy link
Contributor

@ingar ingar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 as per our discussion

@hannahhoward hannahhoward merged commit d9d9a15 into master Jul 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants