Benchmark Report #463

shubham3121 · 2024-12-04T07:58:26Z

This pull request introduces a comprehensive benchmarking framework for the SyftBox client. We measure the following:

Network Stats:
- We measure HTTP Stats like dns, ssl handshake time, etc.
- We measure TCP Stats which measure the TCP latency and jitter in making connection to the server url.
Sync Stats:
- We measure the time it takes to upload/download a sample file to the syftbox server
- We benchmark it against different file sizes

Implemented different report types:

JSONReport - JSON formatted benchmark report, available in a json file.
ConsoleReport - Human readable format benchmark report, available on stdout

Example

syftbox client benchmark

Checklist

I have followed the Contribution Guidelines and Code of Conduct
I have commented my code following the OpenMined Styleguide
I have labeled this PR with the relevant Type labels
My changes are covered by tests

…tors - integrate benchmark to cli: `syftbox client benchmark`

…easurement

- add base classes for Report Generation and MetricData - adapt network metric to use dataclasses

- remove run_benchmark and init benchmark in cli itself. - remove option to provide n_runs in benchmark, alway defaults to DEFAULT_BENCHMARK_RUNS - update HTTPStatsPerf and TCPStatsPerf to return dataclasses instead of dict

Sync Benchmark Metrics

- add tests for benchmark runner

itstauq

Looks awesome to me 🚀

itstauq · 2024-12-18T12:11:34Z

syftbox/client/benchmark/netstats_tcp.py

+        """Remove requests older than 1 minute"""
+        cutoff = datetime.now() - timedelta(minutes=1)
+        while self.requests and self.requests[0] < cutoff:
+            self.requests.popleft()


syftbox/client/benchmark/netstats_http.py

- calculate sync reports in case of partial failures - track success rate for sync ops - update tests

shubham3121 added 5 commits December 4, 2024 13:21

add class to measure HTTP and TCP stats for a given url

f8183a0

base class for metric collection

0eba79a

class to measure network performance for server

88999d4

benchmark class to collect benchmark stats from various metric collec…

ab13cfa

…tors - integrate benchmark to cli: `syftbox client benchmark`

Merge branch 'main' into shubham/client-metrics

eda1292

shubham3121 requested a review from yashgorana December 5, 2024 10:05

Merge branch 'main' into shubham/client-metrics

1a4b318

shubham3121 requested a review from rasswanth-s December 5, 2024 10:08

yashgorana and others added 16 commits December 9, 2024 13:56

Merge branch 'main' into shubham/client-metrics

fcfc3e8

add dependency

0b419e5

Implement rate limiting and connection tracking for TCP performance m…

0e992c1

…easurement

Refactor HTTP performance measurement to use curl_cffi

0ef041b

replace pycurl with curl-cffi

71846b1

move benchmark files to a sub folder

f21931f

- add base classes for Report Generation and MetricData - adapt network metric to use dataclasses

decouple report generation logic from metrics collection

a41398d

- remove run_benchmark and init benchmark in cli itself. - remove option to provide n_runs in benchmark, alway defaults to DEFAULT_BENCHMARK_RUNS - update HTTPStatsPerf and TCPStatsPerf to return dataclasses instead of dict

fix typo

48a6246

Merge branch 'main' into shubham/client-metrics

c846a4e

add curl_cffi to pyproject

6d3aba7

Merge branch 'main' into shubham/client-metrics

3112457

add sync metrics to benchmark upload and download of files

1d8725b

integrate sync metric to benchmark report

a6032ca

Merge branch 'shubham/client-metrics' into shubham/sync-metrics

c547d1a

Merge pull request #504 from OpenMined/shubham/sync-metrics

91fe12c

Sync Benchmark Metrics

Merge branch 'main' into shubham/client-metrics

9ef4668

shubham3121 changed the title ~~[WIP] Benchmark Report~~ Benchmark Report Dec 12, 2024

shubham3121 self-assigned this Dec 12, 2024

shubham3121 and others added 4 commits December 12, 2024 23:04

add test for client cli benchmark

54aaef4

- add tests for benchmark runner

add tests for benchmark network and sync metric collectors

4fc8602

add test coverage for benchmark report

9b8b2c0

Merge branch 'main' into shubham/client-metrics

4f8d8d7

yashgorana and others added 9 commits December 15, 2024 15:37

move metrics to client

8432263

move base classes to __init__

0381d61

some renames

ab795dc

cleanup http timings

818ffd9

clean up sync benchmark

07197ea

fix tests for benchmark

3cffa3c

Merge branch 'main' into shubham/client-metrics

4e8944d

fix passing filename as string to delete file

83675c3

remove print statements

3e33e25

itstauq approved these changes Dec 18, 2024

View reviewed changes

shubham3121 added 5 commits December 18, 2024 19:18

Enhance benchmark reporting with success rate and progress indication

aa111a4

- calculate sync reports in case of partial failures - track success rate for sync ops - update tests

Fix comment numbering in HTTP performance statistics

e46168c

fix tests

5e604c8

Merge branch 'main' into shubham/client-metrics

3134b7d

fix mypy

ac8700f

shubham3121 merged commit 0379c42 into main Dec 18, 2024
14 checks passed

itstauq deleted the shubham/client-metrics branch December 18, 2024 20:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark Report #463

Benchmark Report #463

shubham3121 commented Dec 4, 2024 •

edited

Loading

itstauq left a comment

itstauq Dec 18, 2024

Benchmark Report #463

Benchmark Report #463

Conversation

shubham3121 commented Dec 4, 2024 • edited Loading

Example

Checklist

itstauq left a comment

Choose a reason for hiding this comment

itstauq Dec 18, 2024

Choose a reason for hiding this comment

shubham3121 commented Dec 4, 2024 •

edited

Loading