Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add js-libp2p perf tests #244

Merged
merged 39 commits into from
Aug 25, 2023
Merged

feat: add js-libp2p perf tests #244

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
13b79e7
feat: add js-libp2p perf tests
maschad Jul 26, 2023
9849455
Merge branch 'master' into perf/js-libp2p
maschad Jul 26, 2023
af75ae8
test: updated dockerfile and makefile
maschad Jul 28, 2023
0579202
feat: updated makefile for perf test runner script for js
maschad Aug 2, 2023
90426db
feat: updated infra local config + README + js perf script
maschad Aug 3, 2023
bd0d912
chore: updated tfstate
maschad Aug 4, 2023
17dafd6
feat: Makefile updates + remove unnecessary tfstate files
maschad Aug 7, 2023
904e086
Merge branch 'master' into perf/js-libp2p
maschad Aug 7, 2023
6f05881
chore: reverted local long_lived default enable + updated terra lock …
maschad Aug 7, 2023
564d423
test: updated permissions + bash script, removed other impls benchmar…
maschad Aug 11, 2023
8f27002
test: updated to latest js protocol perf
maschad Aug 14, 2023
0832199
test: undo changes for remote run
maschad Aug 14, 2023
c12b1e2
Update perf/runner/src/index.ts
mxinden Aug 15, 2023
f0a6402
feat: updated the Makefile and executable for faster builds
maschad Aug 15, 2023
670bcd3
feat: updated the Makefile and executable for faster builds
maschad Aug 15, 2023
ff9525a
test: updated benchmarks
maschad Aug 16, 2023
0a220bf
test: install node on EC2 instance in Makefile
maschad Aug 16, 2023
1f6e88b
chore: restore perf runner defaults
maschad Aug 16, 2023
cf93fac
chore: restore benchmark results + untrack local terraform lock file
maschad Aug 16, 2023
8c6f0bc
chore: update js-libp2p gitignore
maschad Aug 16, 2023
7a4395b
Merge branch 'master' into perf/js-libp2p
maschad Aug 16, 2023
e0f9973
feat: added js-libp2p version to versions
maschad Aug 16, 2023
b49501e
Update perf/impl/js-libp2p/v0.46/Makefile
mxinden Aug 18, 2023
32f872a
chore: re-introduce terraform lockfile
maschad Aug 18, 2023
bbc315a
Merge branch 'master' into perf/js-libp2p
maschad Aug 18, 2023
f479d73
chore: clean up Makefile
maschad Aug 18, 2023
e7aee86
chore: use yum to install node js on EC2 in terraform user data
maschad Aug 18, 2023
1a8a9b0
Merge branch 'master' into perf/js-libp2p
maschad Aug 18, 2023
ba3c508
chore: add user_data_replace_on_change variable and set it to true
maschad Aug 19, 2023
aa03eee
Revert "chore: add user_data_replace_on_change variable and set it to…
mxinden Aug 21, 2023
daae7f9
Revert "chore: use yum to install node js on EC2 in terraform user data"
mxinden Aug 21, 2023
0105b09
Merge branch 'master' into perf/js-libp2p
maschad Aug 21, 2023
8e528c0
chore: ensure perf servers are shutdown in tests
maschad Aug 21, 2023
48f3c13
chore: restore to 100 pings
maschad Aug 21, 2023
8f8b0ce
Revert "chore: ensure perf servers are shutdown in tests"
mxinden Aug 22, 2023
6d4c137
fix(perf/js-libp2p/perf): kill child process (node) when killed
mxinden Aug 22, 2023
8381f2e
fix(.gitignore): newline symbol
mxinden Aug 23, 2023
df2f636
perf: update benchmark results
mxinden Aug 23, 2023
4d95675
Merge branch 'master' into perf/js-libp2p
maschad Aug 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion perf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ _WARNING_: Running the perf tests might take a while.
- `--run-server`
- Running as a libp2p-perf client
- Input via command line
- `--server-ip-address`
- `--server-address`
maschad marked this conversation as resolved.
Show resolved Hide resolved
- `--transport` (see `runner/versions.ts` for possible variants)
- `--upload-bytes` number of bytes to upload per stream.
- `--download-bytes` number of bytes to download per stream.
Expand Down
27 changes: 15 additions & 12 deletions perf/impl/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,25 @@ RUST_SUBDIRS := $(wildcard rust-libp2p/*/.)
RUST_QUINN_SUBDIRS := $(wildcard rust-libp2p-quinn/*/.)
HTTPS_SUBDIRS := $(wildcard https/*/.)
QUIC_GO_SUBDIRS := $(wildcard quic-go/*/.)
JS_SUBDIRS := $(wildcard js-libp2p/*/.)

all: $(RUST_SUBDIRS) $(RUST_QUINN_SUBDIRS) $(GO_SUBDIRS) $(HTTPS_SUBDIRS) $(QUIC_GO_SUBDIRS)
$(RUST_SUBDIRS):
$(MAKE) -C $@
$(RUST_QUINN_SUBDIRS):
$(MAKE) -C $@
$(GO_SUBDIRS):
$(MAKE) -C $@
$(HTTPS_SUBDIRS):
$(MAKE) -C $@
$(QUIC_GO_SUBDIRS):
all: $(RUST_SUBDIRS) $(RUST_QUINN_SUBDIRS) $(GO_SUBDIRS) $(HTTPS_SUBDIRS) $(QUIC_GO_SUBDIRS) $(JS_SUBDIRS)
# $(RUST_SUBDIRS):
# $(MAKE) -C $@
# $(RUST_QUINN_SUBDIRS):
# $(MAKE) -C $@
# $(GO_SUBDIRS):
# $(MAKE) -C $@
# $(HTTPS_SUBDIRS):
# $(MAKE) -C $@
# $(QUIC_GO_SUBDIRS):
# $(MAKE) -C $@
$(JS_SUBDIRS):
$(MAKE) -C $@

clean: $(RUST_SUBDIRS:%=%clean) $(RUST_QUINN_SUBDIRS:%=%clean) $(GO_SUBDIRS:%=%clean) $(HTTPS_SUBDIRS:%=%clean) $(QUIC_GO_SUBDIRS:%=%clean)
clean: $(RUST_SUBDIRS:%=%clean) $(RUST_QUINN_SUBDIRS:%=%clean) $(GO_SUBDIRS:%=%clean) $(HTTPS_SUBDIRS:%=%clean) $(QUIC_GO_SUBDIRS:%=%clean) $(JS_SUBDIRS:%=%clean)

%clean:
$(MAKE) -C $* clean

.PHONY: $(RUST_SUBDIRS) $(RUST_QUINN_SUBDIRS) $(GO_SUBDIRS) $(HTTPS_SUBDIRS) $(QUIC_GO_SUBDIRS) all clean
.PHONY: $(RUST_SUBDIRS) $(RUST_QUINN_SUBDIRS) $(GO_SUBDIRS) $(HTTPS_SUBDIRS) $(QUIC_GO_SUBDIRS) $(JS_SUBDIRS) all clean
5 changes: 5 additions & 0 deletions perf/impl/js-libp2p/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
js-libp2p-*.zip
js-libp2p-*
js-libp2p-*/*
image.json
perf-tests
19 changes: 19 additions & 0 deletions perf/impl/js-libp2p/v0.45/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
commitSha := 3345f28b3b13fbe6b4e333466488e9d0bc677322

all: perf

perf: js-libp2p-${commitSha}
docker run --rm -v "$(shell pwd)/js-libp2p-${commitSha}":/usr/src/myapp -w /usr/src/myapp node:19.7.0 npm install
docker run --rm -v "$(shell pwd)/js-libp2p-${commitSha}":/usr/src/myapp -w /usr/src/myapp node:19.7.0 npm run build

js-libp2p-${commitSha}: js-libp2p-${commitSha}.zip
unzip -o js-libp2p-${commitSha}.zip

js-libp2p-${commitSha}.zip:
wget -O $@ "https://github.com/libp2p/js-libp2p/archive/${commitSha}.zip"

clean:
rm -rf js-libp2p-*.zip
rm -rf js-libp2p-*
mxinden marked this conversation as resolved.
Show resolved Hide resolved

.PHONY: all clean perf
46 changes: 46 additions & 0 deletions perf/impl/js-libp2p/v0.45/perf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash

# Find the path to the Node.js executable
node_path=$(which node)

# Check if Node.js is installed
if [ -z "$node_path" ]; then
echo "Node.js is not installed or not in the PATH."
# Update package list and install Node.js
sudo yum update -y
sudo yum install -y gcc-c++ make
curl -fsSL https://deb.nodesource.com/19.x | sudo -E bash -
sudo yum install -y nodejs

fi

run_server=false
server_address=""
upload_bytes=0
download_bytes=0
transport=""

# Parse named parameters manually
for ((i = 1; i <= $#; i++)); do
if [ "${!i}" == "--server-address" ]; then
server_address="${@:i+1:1}"
fi
if [ "${!i}" == "--upload-bytes" ]; then
upload_bytes="${@:i+1:1}"
fi
if [ "${!i}" == "--download-bytes" ]; then
download_bytes="${@:i+1:1}"
fi
if [ "${!i}" == "--transport" ]; then
transport="${@:i+1:1}"
fi
if [ "${!i}" == "--run-server" ]; then
run_server=true
fi
done

echo "server_address: $server_address"

# Run perf
node impl/js-libp2p/v0.45/js-libp2p-3345f28b3b13fbe6b4e333466488e9d0bc677322/packages/protocol-perf/dist/src/main.js --run-server=$run_server --server-address=$server_address --upload-bytes=$upload_bytes --download-bytes=$download_bytes --transport=$transport
# node js-libp2p-3345f28b3b13fbe6b4e333466488e9d0bc677322/packages/protocol-perf/dist/src/main.js --run-server=$run_server --server-address=$server_address --upload-bytes=$upload_bytes --download-bytes=$download_bytes --transport=$transport
4 changes: 2 additions & 2 deletions perf/runner/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ async function main(clientPublicIP: string, serverPublicIP: string) {
function runPing(clientPublicIP: string, serverPublicIP: string): PingResults {
console.error(`= run 100 pings from client to server`);

const cmd = `ssh -o StrictHostKeyChecking=no ec2-user@${clientPublicIP} 'ping -c 100 ${serverPublicIP}'`;
const cmd = `ssh -o StrictHostKeyChecking=no ec2-user@${clientPublicIP} 'ping -c 1 ${serverPublicIP}'`;
maschad marked this conversation as resolved.
Show resolved Hide resolved
const stdout = execCommand(cmd).toString();

// Extract the time from each ping
Expand All @@ -72,7 +72,7 @@ function runPing(clientPublicIP: string, serverPublicIP: string): PingResults {
}

function runIPerf(clientPublicIP: string, serverPublicIP: string): IperfResults {
const iterations = 60;
const iterations = 1;
console.error(`= run ${iterations} iPerf TCP from client to server`);

const killCMD = `ssh -o StrictHostKeyChecking=no ec2-user@${serverPublicIP} 'kill $(cat pidfile); rm pidfile; rm server.log || true'`;
Expand Down
89 changes: 57 additions & 32 deletions perf/runner/src/versions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,39 +5,64 @@ export type Version = {
}

export const versions: Array<Version> = [
// {
maschad marked this conversation as resolved.
Show resolved Hide resolved
// id: "v0.34",
// implementation: "quic-go",
// transportStacks: ["quic-v1"]
// },
// {
// id: "v0.52",
// implementation: "rust-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
// {
// id: "v0.52",
// implementation: "rust-libp2p-quinn",
// transportStacks: ["quic-v1"]
// },
// {
// id: "v0.1",
// implementation: "https",
// transportStacks: ["tcp"]
// },
// {
// id: "v0.27",
// implementation: "go-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
// {
// id: "v0.28",
// implementation: "go-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
// {
// id: "v0.29",
// implementation: "go-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
{
id: "v0.34",
implementation: "quic-go",
transportStacks: ["quic-v1"]
},
{
id: "v0.52",
implementation: "rust-libp2p",
transportStacks: ["tcp", "quic-v1"]
},
{
id: "master",
implementation: "rust-libp2p",
transportStacks: ["tcp", "quic-v1"]
},
{
id: "v0.1",
implementation: "https",
id: "v0.45",
implementation: "js-libp2p",
transportStacks: ["tcp"]
},
{
id: "v0.27",
implementation: "go-libp2p",
transportStacks: ["tcp", "quic-v1"]
},
{
id: "v0.28",
implementation: "go-libp2p",
transportStacks: ["tcp", "quic-v1"]
},
{
id: "v0.29",
implementation: "go-libp2p",
transportStacks: ["tcp", "quic-v1"]
},
// {
// id: "v0.34",
// implementation: "quic-go",
// transportStacks: ["quic-v1"]
// },
// {
// id: "v0.52",
// implementation: "rust-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
// {
// id: "master",
// implementation: "rust-libp2p",
// transportStacks: ["tcp", "quic-v1"]
// },
// {
// id: "v0.1",
// implementation: "https",
// transportStacks: ["tcp"]
// }
]
30 changes: 14 additions & 16 deletions perf/terraform/configs/local/.terraform.lock.hcl
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion perf/terraform/configs/local/terraform.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ variable "ci_enabled" {
variable "long_lived_enabled" {
type = bool
description = "Whether or not to create long lived resources (in CI, used across runs; e.g. VPCs)"
default = false
default = true
maschad marked this conversation as resolved.
Show resolved Hide resolved
}

variable "short_lived_enabled" {
Expand Down
2 changes: 1 addition & 1 deletion perf/terraform/modules/short_lived/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ resource "aws_instance" "perf" {

launch_template {
name = "perf-node"
version = "3"
version = "1"
maschad marked this conversation as resolved.
Show resolved Hide resolved
}

key_name = aws_key_pair.perf.key_name
Expand Down
Loading