Dev0.4.0 vendor exp #1962
Closed
Dev0.4.0 vendor exp #1962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
License: MIT Signed-off-by: David Dias <daviddias.p@gmail.com>
This used to lead to large refcount numbers, causing Flush to create a lot of IPFS objects, and merkledag to consume tens of gigabytes of RAM. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
OS X sed is documented as "-i SUFFIX", GNU sed as "-iSUFFIX". The one consistent case seems to be "-iSUFFIX", where suffix cannot empty (or OS X will parse the next argument as the suffix). This used to leave around files named `refsout=` on Linux, and was just confusing. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
These secondary copies were never actually queried, and didn't contain the indirect refcounts so they couldn't become the authoritative source anyway as is. New goal is to move pinning into IPFS objects. A migration will be needed to remove the old data from the datastore. This can happen at any time after this commit. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
Pinner had method GetManual that returned a ManualPinner, so every Pinner had to implement ManualPinner anyway. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
Platform-dependent behavior is not nice, and negative refcounts are not very useful. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> sharness: Don't assume we know all things that can create garbage License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
WARNING: No migration performed! That needs to come in a separate commit, perhaps amended into this one. This is the minimal rewrite, only changing the storage from JSON(+extra keys) in Datastore to IPFS objects. All of the pinning state is still loaded in memory, and written from scratch on Flush. To do more would require API changes, e.g. adding error returns. Set/Multiset is not cleanly separated into a library, yet, as it's API is expected to change radically. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
…lure License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
There was doublewrapping with an unneeded msgio. given that we use a stream muxer now, msgio is only needed by secureConn -- to signal the boundaries of an encrypted / mac-ed ciphertext. Side note: i think including the varint length in the clear is actually a bad idea that can be exploited by an attacker. it should be encrypted, too. (TODO) License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
* ID service stream * make the relay service use msmux * fix nc tests Note from jbenet: Maybe we should remove the old protocol/muxer and see what breaks. It shouldn't be used by anything now. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
The addition of a locking interface to the blockstore allows us to perform atomic operations on the underlying datastore without having to worry about different operations happening in the background, such as garbage collection. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
This commit improves (fixes) the FetchGraph call for recursively fetching every descendant node of a given merkledag node. This operation should be the simplest way of ensuring that you have replicated a dag locally. This commit also implements a method in the merkledag package called EnumerateChildren, this method is used to get a set of the keys of every descendant node of the given node. All keys found are noted in the passed in KeySet, which may in the future be implemented on disk to avoid excessive memory consumption. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> dont GC blocks used by pinner License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> comment GC algo License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> add lock to blockstore to prevent GC from eating wanted blocks License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> improve FetchGraph License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> separate interfaces for blockstore and GCBlockstore License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> reintroduce indirect pinning, add enumerateChildren dag method License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
…m for S3 License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
…"private" License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
To test it, set up an S3 bucket (in an AWS region that is not US
Standard, for read-after-write consistency), run `ipfs init`, then
edit `~/.ipfs/config` to say
"Datastore": {
"Type": "s3",
"Region": "us-west-1",
"Bucket": "mahbukkit",
"ACL": "private"
},
with the right values. Set `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` in the environment and you should be able to
run `ipfs add` and `ipfs cat` and see the bucket be populated.
No automated tests exist, unfortunately. S3 is thorny to simulate.
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
Last argument was dropped in ffd4c3f License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
Without this, all entries will have nlink==0, which confuses a bunch of tools. Most dramatically, systemd-nspawn enters a busy loop in its lock utility function. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
This used to cause files e.g. being edited with `vi` to become 0-size. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
Callers assume this is safe to call whenever, let's make it so. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: rht <rhtbot@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>
whyrusleeping
force-pushed
the
dev0.4.0-vendor-exp
branch
from
a45211f
to
98dce90
Compare
|
@whyrusleeping which are the relevant extracted out repos? One important thing that made npm very successful is that it tries hard to get a repo associated with the packages. that way users can find where to contribute + file bugs, etc. would be good to have properties like also, can we put all of them in the ipfs org? that way we'll be sure they stay there and dont change names long term etc. |
|
re: the issues key, thats a good idea.
We can, i'm just leery about cluttering up the ipfs org with all of these packages. maybe a |
why is a large number of repos a problem? they're not hurting anyone-- there's a search bar and already way too many to browse. the directory is what people should look at for descriptions anyway |
|
there's a bunch of problems with managing multiple orgs, acl's on github are annoying. |
ghost
added
the
whyrusleeping
force-pushed
the
dev0.4.0
branch
2 times, most recently
from
68b9745
to
b0a8591
Compare
whyrusleeping
force-pushed
the
dev0.4.0
branch
from
cf514d8
to
3224ae0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR begins the process of using gx and the go vendor experiment to break code out from the main go-ipfs codebase. Getting this in will make pulling out (and vendoring) libp2p much easier.