Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract dagservice and friends from go-ipfs #8

Merged
merged 21 commits into from
Dec 11, 2017

Commits on Oct 16, 2017

  1. nit: document Node

    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    ac092e5 View commit details
    Browse the repository at this point in the history
  2. [WIP] [RFC] extract dagservice and friends from go-ipfs

    This is a WIP/RFC attempt at extracting DAGService from go-ipfs.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    b3a1f4b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0f9e9ed View commit details
    Browse the repository at this point in the history
  4. replace LinkService with an optional LinkGetter interface

    This way, not *all* DAGServices need to implement this interface, they can just
    implement it as an optimization.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    2696405 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5837bec View commit details
    Browse the repository at this point in the history
  6. NodePromise: replace interface with concrete type

    Also:
    
    1. Specify the threading guarantees.
    2. Vastly simplify it to use a single channel for synchronization.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    c0311d7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    339e9ea View commit details
    Browse the repository at this point in the history
  8. remove TODO for NodeGetter

    We'll just live with this name.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    5bc8a07 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    30aecc4 View commit details
    Browse the repository at this point in the history
  10. Make DAGService.Remove take a CID.

    We're going to want to do this eventually and we have to refactor anyways so we
    might as well do this now.
    
    Fixes ipfs/kubo#4010
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    2f52265 View commit details
    Browse the repository at this point in the history
  11. dag: move GetMany from DAGService to NodeGetter

    This will allow many consumers of `DAGService` to take `NodeGetter` instead and
    implementing `GetMany` for all `NodeGetter`s is pretty trivial.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    3380389 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    f2fc6ce View commit details
    Browse the repository at this point in the history
  13. move GetLinks to daghelpers

    Keep merkledag clean.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    07869d6 View commit details
    Browse the repository at this point in the history
  14. Get rid of OfflineNodeGetter

    It really just doesn't fit. We're working on making this method obsolete
    anyways.
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    5225978 View commit details
    Browse the repository at this point in the history
  15. port async batch commit code from ipfs

    (ipfs/kubo#4296)
    
    1. Modern storage devices (i.e., SSDs) tend to be highly parallel.
    2. Allows us to read and write at the same time (avoids pausing while flushing).
    
    fixes ipfs/kubo#898 (comment)
    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    1af7e81 View commit details
    Browse the repository at this point in the history
  16. add a basic test for Batch

    Stebalien committed Oct 16, 2017
    Configuration menu
    Copy the full SHA
    0408f8d View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2017

  1. Make remove idempotent.

    1. Add is already idempotent. This makes remove match.
    2. Generally, all we care about is that the node no longer exists in the DAG. If
    two callers remove a node at the same time, they should both succeed. Currently,
    we *ignore* the result of Remove in go-ipfs. IMO, it would be better to let
    errors be *errors* and only return an error if something goes wrong.
    3. This can be significantly faster. It allows us to batch/queue removes (as
    long as we guarantee that they'll eventually happen).
    4. This matches how most databases/key-value stores operate.
    
    An alternative would be to return `(deleted bool, err error)` but then we don't
    get the speedup.
    Stebalien committed Nov 15, 2017
    Configuration menu
    Copy the full SHA
    6f9115b View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2017

  1. don't return CIDs on add

    The caller can just call `node.Cid()` and returning CIDs from `AddMany` requires
    allocation.
    Stebalien committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    b403323 View commit details
    Browse the repository at this point in the history
  2. Add RemoveMany method

    Stebalien committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    f10b5dd View commit details
    Browse the repository at this point in the history
  3. add contexts to Add/Remove methods

    We'll need these for slower/remote datastores.
    Stebalien committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    44a7801 View commit details
    Browse the repository at this point in the history
  4. add context to batch

    I considered (well, implemented then threw it away) allowing contexts on all
    calls to Batch (Add, Commit, etc). However, really, you should treat a batch
    as a single large "operation".
    
    I also went down the road of generalizing batches to sessions. However, it
    became immediately obvious that permitting add *and* remove *and* fetch would
    require a lot of bookkeeping and that you'd lose a lot of performance. So, we'll
    do that separately.
    Stebalien committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    6cf32cc View commit details
    Browse the repository at this point in the history