Skip to content
This repository has been archived by the owner on Feb 1, 2023. It is now read-only.

feat(Benchmarks): Add real world dup blocks test #25

Merged
merged 1 commit into from
Nov 15, 2018

Conversation

hannahhoward
Copy link
Contributor

@hannahhoward hannahhoward commented Nov 14, 2018

Goals

Build a benchmark that can replicate the problems that #8 has with LS-ing Wikipedia

Implementation

  • add a delay generator that similates real world latencies one might encounter on the internet
  • modify virtual network to accept different latencies for different
    peers based on using NextWaitTime on passed delay
  • modify dup_blocks_test subtestDistributeAndFetch to accept a custom
    delay
  • Add a real world benchmarks that simulates the kinds of problems one might
    encounter bitswaping with a long lived session and a large swarm of
    peers with real world latency distributions (that causes [WIP] Work towards improving the duplicate blocks issue #8 not to
    function well in practice)

For Discussion

The goal here is to have a good benchmark to compare against as we continue to develop improvements to bitswap.

One thing not addressed here are transfer speeds (though you can think of latency in this test as really being time to first byte -- i.e. traditional latency -- + download time). Is that worth addressing seperately.

Benchmarks with results posted in comment.

child of ipfs/kubo#5723

@ghost ghost assigned hannahhoward Nov 14, 2018
@ghost ghost added the status/in-progress In progress label Nov 14, 2018
@hannahhoward
Copy link
Contributor Author

Benchmark Results master vs feat/reduce-dupes

These are comparisons of each of the three network conditions on both master and Jeremy's branch.

Super great network

Everyone’s coming in around 60ms latency

Branch Duplicate Blocks Total Time
master 52804 29.317s
feat/reduce-dupes 1192 7.563s

Jeremy PR kicks butt cause it makes less requests / doesn't
deal have to deal with dupes

So So Network

40% at 60ms latency, 30% at 200ms latency, 30% at 800ms latency

Branch Duplicate Blocks Total Time
master 30704 16.599s
feat/reduce-dupes 1192 33.628s

Master gets faster because it doesn't deal with as many dupes
Meanwhile Jeremy PR still no extra dupes but now quite a bit slower than
master

Worst case scenario (a.l.a. Wikipedia)

40% at 60ms latency, 30% at 200ms latency, 30% at 4000ms latency

Branch Duplicate Blocks Total Time
master 38885 7.066s
feat/reduce-dupes 20045 72.017s

Now Jeremy's PR is getting comparable dupes due to broadcasts.
Moreover, it's now almost 10x slower than master

@hannahhoward
Copy link
Contributor Author

I would be curious on @whyrusleeping 's feedback if this seems like a good test.

@hannahhoward
Copy link
Contributor Author

Seeking 1 LGTM from anyone to merge, since this is just benchmarks

- add a delay generator that similates real world latencies one might encounter on the internet
- modify virtual network to accept different latencies for different
peers based on using NextWaitTime on passed delay
- modify dup_blocks_test subtestDistributeAndFetch to accept a custom
delay
- Add a real world benchmarks that simulates the kinds of problems one might
encounter bitswaping with a long lived session and a large swarm of
peers with real world latency distributions (that causes #8 not to
function well in practice)
@hannahhoward hannahhoward force-pushed the feat/dup-blocks-test-enhancement branch from b438f95 to 39fa3c7 Compare November 14, 2018 21:20
Copy link
Contributor

@michaelavila michaelavila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hannahhoward hannahhoward merged commit edf2496 into master Nov 15, 2018
@ghost ghost removed the status/in-progress In progress label Nov 15, 2018
Jorropo pushed a commit to Jorropo/go-libipfs that referenced this pull request Jan 26, 2023
…enhancement

feat(Benchmarks): Add real world dup blocks test

This commit was moved from ipfs/go-bitswap@edf2496
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants