Massive DHT traffic on adding big datasets #2828

Kubuxu · 2016-06-09T15:18:25Z

@magik6k was trying to add cdnjs (something I did few months ago) he have hit #2823 but also reported that adding the dataset which was about 21GB created about 3TB of traffic.

@magik6k could you give more precise data on this?

whyrusleeping · 2016-06-12T20:38:17Z

The solution i'm investigating here is batching together outgoing provide operations, combining findpeer queries and outgoing 'puts'. The issue is that it might require making some backwards incompatible changes to the dht protocol, which isnt really a huge problem since we have multistream

magik6k · 2016-06-12T21:01:25Z

Network: 800 GiB Out, 2 TiB in
While adding files it was using ~50/100Mbps
Around 10-20 Files/s, depending on file size, usually around 10-100KB
2633788 files
2-3 days IIRC

whyrusleeping · 2016-06-12T21:03:46Z

@magik6k Thanks! I will keep you posted as we make improvements to this.

parkan · 2016-08-19T17:04:41Z

Similar report, while adding ~70,000 objects (~100k ea) we were maxing out our instance's traffic (70MBit in, 160MBit out) for over 14 hours (!!!), at which point someone killed the IPFS daemon.

Back of the envelope, this is at least 1TB outgoing for ~10GB files.

One particular problem: because the daemon was killed before the process completed, it seems that almost none of the files were pinned. Running ipfs repo gc deleted almost everything:

; before gc
NumObjects       2995522
RepoSize         20
RepoPath         /home/ubuntu/.ipfs
Version          fs-repo@4

; after gc
NumObjects       771
RepoSize         20
RepoPath         /home/ubuntu/.ipfs
Version          fs-repo@4

(side note, the chunk counts seem way too high too, almost 40x per image instead of expected 4x)

whyrusleeping · 2016-08-19T17:23:52Z

@parkan which version of ipfs are you using? This is definitely a known issue, but it has been improved slightly since 0.4.2

parkan · 2016-08-19T17:25:08Z

@whyrusleeping 0.4.3-rc3

parkan · 2016-08-19T17:26:16Z

@whyrusleeping my intuition is that there's some kind of context that's not being released before a queue is drained, which is supported by none of the files being pinned successfully after killing the daemon. Does this seem plausible?

whyrusleeping · 2016-08-19T17:27:04Z

@parkan are you killing the daemon before the add call is complete?

parkan · 2016-08-19T17:27:36Z

@whyrusleeping no, these are all individual add calls that complete reasonably quickly, load/network traffic happens w/o client interaction

jbenet · 2016-08-19T17:41:00Z

One particular problem: because the daemon was killed before the process completed, it seems that almost none of the files were pinned. Running ipfs repo gc deleted almost everything:

That's a very serious bug to figure out. consistency is paramout here and that's much more important to get right.

Similar report, while adding ~70,000 objects (~100k ea) we were maxing out our instance's traffic (70MBit in, 160MBit out) for over 14 hours (!!!), at which point someone killed the IPFS daemon.

Back of the envelope, this is at least 1TB outgoing for ~10GB files.

This is caused by providing random access to all content, as discussed via email. for the large use case, we should look at recompiling go-ipfs without creating a provider record for each object, and leverage direct connections. (FWIW, orbit on js-ipfs works without the DHT entirely, and i imagine your use case here could do very well if you make sure to connect the relevant nodes together so there's no discovery to happen. This is the way to do it before pubsub connects them for you).

whyrusleeping · 2016-08-19T17:41:03Z

You can check which objects are pinned after an add with: ipfs pin ls --type=recursive.

It also seems really weird that your repo size is listed as 20 for both of those calls...

whyrusleeping · 2016-08-19T17:42:25Z

@parkan also, what is the process for adding files? is it just a single call to ipfs add ? or many smaller calls broken up? Are you passing any other flags?

jbenet · 2016-08-19T17:42:39Z

@parkan on chunk size, i'd be interested (separately) how well Rabin Fingerprint based chunking (ipfs add -s <file>) does for your images. If they're similar it should reduce the storage, but this is experimental and not proved to help. it's also slower on add, as it's got to do math when chunking.

parkan · 2016-08-19T17:43:59Z

@whyrusleeping ipfs pin ls was showing <500 objects pinned successfully, I don't think any of them are from this add attempt

also, small correction to my previous statement: not all ipfs add calls had completed, basically it was like this:

we add ~10k files with relatively fast return from ipfs add
adds slow down, taking >30s per file
daemon is killed

parkan · 2016-08-19T17:45:48Z

@whyrusleeping these are many separate adds, we basically write a temporary file (incidentally, being able to pin from stdin would be great), ipfs add it (no options) and wait for return

whyrusleeping · 2016-08-19T17:46:07Z

@parkan Ah, yeah. The pins arent written until the add is complete. An ipfs add call will only pin the root hash of the result of any given call to ipfs add.

whyrusleeping · 2016-08-19T17:47:06Z

being able to pin from stdin would be great

➜  ~ echo QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n | ipfs pin add
pinned QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n recursively

whyrusleeping · 2016-08-19T17:48:34Z

When the adds slow down that far could you get me some of the performance debugging info from https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md ?

parkan · 2016-08-19T17:51:02Z

@whyrusleeping sorry, maybe I was being unclear: for 10k+ files the "add" part was completed (i.e. the call to ipfs add returned) but pin did not land.

re: stdin, I meant ipfs add, not ipfs pin add :) Can I just pipe binary data into ipfs add -?

whyrusleeping · 2016-08-19T17:56:15Z

So, you did:

ipfs add -r <dir with 100k files in it>
...
...
...
added <some hash> <that dirs name>

Right?
Does that final hash from that add call show up in the output of ipfs pin ls --type=recursive ?

I'm also confused about what is meant by 'adds slow down' if the add call completed by that point. Was this a subsequent call to ipfs add with more files?

re: stdin: you can indeed just pipe binary data to ipfs add :)

➜  ~ echo "hello" | ipfs add
added QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
➜  ~ dd if=/dev/urandom bs=4M count=2 | ipfs add
 7.75 MB / ? [----------=---------------------------------------------------------] 2+0 records in
2+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.571907 s, 14.7 MB/s
added QmbPmPSxWcQqJGcq7hHJ7CGMxCQ2zxpMpNHLxARePPNv6n QmbPmPSxWcQqJGcq7hHJ7CGMxCQ2zxpMpNHLxARePPNv6n

parkan · 2016-08-19T17:56:27Z

here's the parsed dump:

crypto/sha256.(*digest).Sum                                                                                         1
runtime.goexit                                                                                                      1
os/signal.signal_recv                                                                                               1
gx/ipfs/QmNQynaz7qfriSUJkiEZUrm2Wen1u3Kj9goZzWtrPyu7XR/go-log.(*MirrorWriter).logRoutine                            1
gx/ipfs/QmV3NSS3A1kX5s28r7yLczhDsXzkgo65cqRgKFXYunWZmD/metrics.init.1.func2                                         1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb/util.(*BufferPool).drain                   1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb.(*DB).mpoolDrain                           1
runtime.gopark                                                                                                      1
main.(*IntrHandler).Handle.func1                                                                                    1
gx/ipfs/QmeYJHEk8UjVVZ4XCRTZe6dFQrb8pGWD81LYCgeLp8CvMB/go-metrics.(*meterArbiter).tick                              1
gx/ipfs/QmduCCgTaLnxwwf9RFQy2PMUytrKcEH9msohtVxSBZUdgu/go-peerstream.(*Swarm).connGarbageCollect                    1
main.daemonFunc.func1                                                                                               1
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess.(*process).CloseAfterChildren                      1
github.com/ipfs/go-ipfs/routing/dht/providers.(*ProviderManager).run                                                1
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/briantigerchow/pubsub.(*PubSub).start                      1
github.com/ipfs/go-ipfs/exchange/bitswap/decision.(*Engine).nextEnvelope                                            1
github.com/ipfs/go-ipfs/exchange/bitswap.(*WantManager).Run                                                         1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb.(*DB).jWriter                              1
github.com/ipfs/go-ipfs/exchange/bitswap.New.func2                                                                  1
github.com/ipfs/go-ipfs/mfs.(*Republisher).Run                                                                      1
github.com/ipfs/go-ipfs/exchange/bitswap.(*Bitswap).providerQueryManager                                            1
github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).Bootstrap.func1                                                      1
github.com/ipfs/go-ipfs/exchange/bitswap.(*Bitswap).rebroadcastWorker                                               1
github.com/ipfs/go-ipfs/exchange/bitswap.(*Bitswap).provideCollector                                                1
github.com/ipfs/go-ipfs/exchange/bitswap.(*Bitswap).provideWorker                                                   1
github.com/ipfs/go-ipfs/namesys/republisher.(*Republisher).Run                                                      1
gx/ipfs/QmSscYPCcE1H3UQr2tnsJ2a9dK9LsHTBGgP71VW6fz67e5/mdns.(*client).query                                         1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb.(*DB).mCompaction                          1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb.(*DB).compactionError                      1
sync.(*Pool).Get                                                                                                    1
github.com/ipfs/go-ipfs/blocks/blockstore.(*blockstore).AllKeysChan.func2                                           1
gx/ipfs/QmTxLSvdhwg68WJimdS6icLPhZi28aTp6b7uihC2Yb47Xk/go-datastore/namespace.(*datastore).Query.func1              1
gx/ipfs/QmTxLSvdhwg68WJimdS6icLPhZi28aTp6b7uihC2Yb47Xk/go-datastore/keytransform.(*ktds).Query.func1                1
gx/ipfs/QmTxLSvdhwg68WJimdS6icLPhZi28aTp6b7uihC2Yb47Xk/go-datastore/query.ResultsWithChan.func1                     1
gx/ipfs/QmTxLSvdhwg68WJimdS6icLPhZi28aTp6b7uihC2Yb47Xk/go-datastore/flatfs.(*Datastore).Query.func1.1               1
gx/ipfs/QmbBhyDKsY4mbY6xsKt3qu9Y7FPvMJ6qbD8AMjYYvPRw1g/goleveldb/leveldb.(*DB).tCompaction                          1
github.com/ipfs/go-ipfs/commands.(*ChannelMarshaler).Read                                                           1
github.com/ipfs/go-ipfs/commands/http.internalHandler.ServeHTTP.func2                                               1
runtime/pprof.writeGoroutineStacks                                                                                  1
gx/ipfs/QmQdnfvZQuhdT93LNc5bos52wAmdr3G2p6G8teLJMEN32P/go-libp2p-peerstore.(*AddrManager).AddAddrs                  1
gx/ipfs/QmVCe3SNMjkcPgnpFhZs719dheq6xE7gJwjzV7aWcUM4Ms/go-libp2p/p2p/discovery.(*mdnsService).pollForEntries.func1  1
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Session).Ping                                        1
github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).GetClosestPeers.func2                                                1
syscall.Syscall                                                                                                     1
main.daemonFunc                                                                                                     1
gx/ipfs/QmVCe3SNMjkcPgnpFhZs719dheq6xE7gJwjzV7aWcUM4Ms/go-libp2p/p2p/net/swarm.(*Swarm).addConnListener.func2       2
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/periodic.callOnTicker.func1                        2
gx/ipfs/QmVCe3SNMjkcPgnpFhZs719dheq6xE7gJwjzV7aWcUM4Ms/go-libp2p/p2p/net/conn.(*listener).Accept                    2
github.com/ipfs/go-ipfs/core/corehttp.Serve                                                                         2
main.merge.func1                                                                                                    2
gx/ipfs/QmdhsRK1EK2fvAz2i2SH5DEfkL6seDuyMYEsxKa9Braim3/client_golang/prometheus.computeApproximateRequestSize       2
gx/ipfs/QmduCCgTaLnxwwf9RFQy2PMUytrKcEH9msohtVxSBZUdgu/go-peerstream.(*Swarm).setupStream.func1                     4
gx/ipfs/Qmf91yhgRLo2dhhbc5zZ7TxjMaR1oxaWaoc9zRZdi1kU4a/go-multistream.(*lazyConn).readHandshake                     6
github.com/ipfs/go-ipfs/exchange/bitswap.(*Bitswap).taskWorker                                                      8
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess.(*process).doClose.func1                           9
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess.(*processLink).AddToChild                          10
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Session).waitForSendErr                              11
gx/ipfs/QmduCCgTaLnxwwf9RFQy2PMUytrKcEH9msohtVxSBZUdgu/go-peerstream.(*Swarm).removeStream.func1                    16
github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).Provide.func1                                                        19
github.com/ipfs/go-ipfs/routing/dht.(*messageSender).ctxReadMsg                                                     25
gx/ipfs/QmX6DhWrpBB5NtadXmPSXYNdVvuLfJXoFNMvUMoVvP5UJa/go-context/io.(*ctxReader).Read                              27
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess.(*process).Close                                   86
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Stream).Read                                         119
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Session).send                                        133
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Session).keepalive                                   143
github.com/ipfs/go-ipfs/exchange/bitswap.(*msgQueue).runQueue                                                       144
gx/ipfs/QmZ8MMKFwA95viWULoSYFZpA4kdFa8idmFSrP12YJwjjaL/yamux.(*Session).AcceptStream                                144
net.runtime_pollWait                                                                                                156
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess.(*process).doClose                                 265
github.com/ipfs/go-ipfs/routing/dht.(*dhtQueryRunner).spawnWorkers                                                  425
gx/ipfs/QmQdnfvZQuhdT93LNc5bos52wAmdr3G2p6G8teLJMEN32P/go-libp2p-peerstore/queue.(*ChanQueue).process.func1         426
gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context.propagateCancel.func1                         426
gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess/context.CloseAfterContext.func1                    433
github.com/ipfs/go-ipfs/routing/dht.(*dhtQueryRunner).Run                                                           509
github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).Provide                                                              511
sync.runtime_Semacquire                                                                                             2187

parkan · 2016-08-19T17:56:57Z

@whyrusleeping no, we are calling ipfs add on each individual file because there's no actual tree to add from

so, again:

for each image
- our script writes a tempfile
- we use py-ipfs-api to ipfs add the file

The slowdown happens about 10k images into the loop

whyrusleeping · 2016-08-19T18:00:47Z

Thanks for the stack dump, that just confirms for me that the problems are caused by the overproviding. The 2000 gorotuines stuck in runtime_Semacquire are somewhat odd though, could I get the full stack dump to investigate that one further?

As for the adds, after each ipfs add call completes (or some subset of them complete) are those hashes visible in the output of ipfs pin ls --type=recursive ? every call to ipfs add should put an entry in there before it returns.

parkan · 2016-08-19T18:03:38Z

@whyrusleeping I'm trying to repro this behavior, but I am quite certain that we ended up with 10,000+ added but not pinned files in yesterday's run

Full stax: https://ipfs.io/ipfs/QmP344ugRiyZRKA2djLgE371MzW2FhwbghnJyH7hmbmAeQ

whyrusleeping · 2018-08-20T23:55:14Z

@casey you don't have to incur any traffic when adding data to ipfs. Adds can be done with the --local option which prevents any outbound DHT traffic during the add.

Also, the outbound traffic usage without that should be far lower than when this issue was created.

cc @magik6k and @Stebalien for other input

Stebalien · 2018-08-21T01:31:51Z

Also, the outbound traffic usage without that should be far lower than when this issue was created.

It should also get much lower in the upcoming release due to libp2p/go-libp2p-kad-dht#182.

casey · 2018-08-21T01:59:46Z

@whyrusleeping Thanks for the response!

If I add data with --local, will other nodes on the network be able to find my node and download the data?

Stebalien · 2018-08-21T04:18:03Z

If I add data with --local, will other nodes on the network be able to find my node and download the data?

Not unless they're already connected to your node, no.

bonedaddy · 2018-08-21T04:32:55Z

How do you invoke the --local option? I took a look at ipfs add --help and there was no --local option

Stebalien · 2018-08-21T04:44:06Z

It's a global flag, which, unfortunately, isn't added to the command-local help. You can find the help by running ipfs --help.

casey · 2018-08-21T04:48:28Z

Unfortunately this doesn't help our use case. Nodes must be able to discover one another in the DHT.

bonedaddy · 2018-08-21T18:17:15Z

@Stebalien ah thank you!

@casey you could look at using a circuit relay that might be able to solve the connectivity issue

casey · 2018-08-21T19:11:28Z

@postables Unfortunately, running relays between nodes is not feasible. We need connectivity between arbitrary nodes, and don't have the resources to expend on dedicated relays.

bonedaddy · 2018-08-21T19:23:13Z

With relay:

Have relay node
Have node running with --local that you add your content to
Direct nodes to your --local node with your relay

Without relay:

Have node not running with --local
Have node running with --local
Upload files to --local node
On non --local node, recursively pin your files from your --local node to your non --local node
Cleanup files on --local node to free up space

Haven't tried this but it should theoretically work in the mean time until these issues are resolved.

casey · 2018-08-21T20:05:57Z

As stated, relay nodes are not feasible. Without relay isn't feasible either, since all nodes are the same. This is a bittorrent-like use case, where all nodes are homogeneous peers, with no nodes having any particular role. As such, it's unclear how to determine which nodes would not use --local, and thus shoulder the burden of the excessive DHT traffic.

bonedaddy · 2018-08-21T20:16:16Z

Not sure how this is unfeasible, all you would need is a single node that runs with --local, and you upload files to that node. You then directly connect your nodes that need the file to that --local node, and pull the data from there which should avoid the excessive DHT traffic.

If none of these options are desirable for you, then IMO IPFS is not suitable for your use case at this time, and it sounds like you should stick with BitTorrent.

casey · 2018-08-21T20:27:29Z

I think that's probably about right. It's unfortunate, since we'd really like to be compatible with the IPFS ecosystem.

bonedaddy · 2018-08-21T20:32:32Z

IPFS is still fairly young, and didn't start gaining wide spread attraction until this year, so you might just have to wait a bit 👍 IPFS has been progressing pretty nicely lately so definitely keep an eye out.

You could also perhaps look into leveraging LibP2P

Stebalien · 2018-08-21T21:58:44Z

Note: We are working on improving the provider (that's what this is called) logic as it obviously doesn't scale. However, it's on the back-burner at the moment while we finish up some of our in-progress endeavors.

bonedaddy · 2018-08-21T22:09:50Z

Do you have a link to the go code for the provider? Would be interesting to take a look at it.

Stebalien · 2018-08-21T22:47:01Z

It's kind of all over the place. This PR tries to improve that a bit: #4333

casey · 2020-09-02T20:01:39Z

Has there been any progress on this? Is this considered to be an issue, or is inserting/looking up each block of all file in the DHT going to be the behavior going forward?

Stebalien · 2020-09-02T20:54:59Z

This is considered an issue. The short term plan is to just announce fewer blocks (requires some changes to bitswap to avoid issues with restarting downlaods). The long-term plan is to introduce additional, more efficient content routing mechanisms.

casey · 2020-09-02T21:00:44Z

I think a simple fix would be to mirror BitTorrent, in the way that the hash of the info dict serves as the "topic" marker. BT clients look up peers via the infohash, and then expect those peers to have blocks. IPFS could do the same thing, use the hash of a "topic" block that contains the hashes, directly or transitively, and expect peers that they find with the topic block hash will have content blocks.

Stebalien · 2020-09-02T21:13:57Z

That's the "announce fewer blocks" plan. However, unlike bittorrent, there isn't really a true "root" hash. The best we can do is use the file root and the root of the request (e.g., use QmFoo /ipfs/QmFoo/path/to/thing.txt). Unfortunately:

Bitswap, the transfer protocol, doesn't know about the relationship between blocks, it just sees blocks. So, if I already have QmFoo, but am missing a block inside thing.txt, it doesn't know to search for QmFoo again. This requires some modifications to pass this extra context to bitswap (planned).
Unfortunately, this kind of change will mean that two slightly different datasets will end up with entirely different "swarms". For example, if I publish wikipedia to IPFS, then update it, the root hashes will be different so peers looking for the new copy will never find peers with the old copy. However, 99% of the files are actually the same.

casey · 2020-09-10T21:10:54Z

My understanding of this is shaky, but if files are split into chunks automatically, then could the hash of the file serve as the topic hash for all the chunks? That would get get us down to one DHT lookup/insert per file.

Kubuxu mentioned this issue Jun 9, 2016

cdnjs ipfs-inactive/archives#35

Open

Kubuxu added topic/dht Topic dht topic/perf Performance labels Jun 12, 2016

Kubuxu mentioned this issue Jun 12, 2016

Leaking goroutines when adding massive amount of files #2823

Closed

whyrusleeping mentioned this issue Jun 12, 2016

a few small changes to make the dht more efficient #2841

Merged

whyrusleeping mentioned this issue Aug 9, 2016

Implement bandwidth limiting #3065

Open

bonedaddy mentioned this issue Aug 21, 2018

IPFS Issue Tracking RTradeLtd/Temporal#95

Open

Stebalien added status/deferred Conscious decision to pause or backlog and removed status/ready Ready to be worked labels Dec 18, 2018

DonaldTsang mentioned this issue May 16, 2019

Performace, or How IPFS will be better than BitTorrent #6342

Closed

Stebalien removed the help wanted Seeking public contribution on this issue label Mar 8, 2020

Stebalien removed this from the Resource Constraints milestone Apr 29, 2020

casey mentioned this issue Jan 27, 2023

Off-chain content ordinals/ord#801

Closed

Massive DHT traffic on adding big datasets #2828

Massive DHT traffic on adding big datasets #2828

Comments

Kubuxu commented Jun 9, 2016

whyrusleeping commented Jun 12, 2016

magik6k commented Jun 12, 2016

whyrusleeping commented Jun 12, 2016

parkan commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

parkan commented Aug 19, 2016

parkan commented Aug 19, 2016 • edited Loading

whyrusleeping commented Aug 19, 2016

parkan commented Aug 19, 2016

jbenet commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

jbenet commented Aug 19, 2016

parkan commented Aug 19, 2016

parkan commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

parkan commented Aug 19, 2016

whyrusleeping commented Aug 19, 2016

parkan commented Aug 19, 2016

parkan commented Aug 19, 2016 • edited Loading

whyrusleeping commented Aug 19, 2016

parkan commented Aug 19, 2016

whyrusleeping commented Aug 20, 2018

Stebalien commented Aug 21, 2018

casey commented Aug 21, 2018

Stebalien commented Aug 21, 2018

bonedaddy commented Aug 21, 2018

Stebalien commented Aug 21, 2018

casey commented Aug 21, 2018

bonedaddy commented Aug 21, 2018

casey commented Aug 21, 2018

bonedaddy commented Aug 21, 2018 • edited Loading

casey commented Aug 21, 2018

bonedaddy commented Aug 21, 2018 • edited Loading

casey commented Aug 21, 2018

bonedaddy commented Aug 21, 2018 • edited Loading

Stebalien commented Aug 21, 2018

bonedaddy commented Aug 21, 2018

Stebalien commented Aug 21, 2018

casey commented Sep 2, 2020

Stebalien commented Sep 2, 2020

casey commented Sep 2, 2020

Stebalien commented Sep 2, 2020

casey commented Sep 10, 2020

parkan commented Aug 19, 2016 •

edited

Loading

parkan commented Aug 19, 2016 •

edited

Loading

bonedaddy commented Aug 21, 2018 •

edited

Loading

bonedaddy commented Aug 21, 2018 •

edited

Loading

bonedaddy commented Aug 21, 2018 •

edited

Loading