Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare air_public_inputs in CI #1394

Merged
merged 8 commits into from
Aug 28, 2023
Merged

Conversation

fmoletta
Copy link
Contributor

@fmoletta fmoletta commented Aug 23, 2023

Depends on #1391

Checklist

  • Linked to Github Issue
  • Unit tests added
  • Integration tests added.
  • This change requires new documentation.
    • Documentation has been added/updated.
    • CHANGELOG has been updated.

@fmoletta fmoletta added the pipelines This PR/issue is exclusively about improving our CI label Aug 23, 2023
@fmoletta fmoletta changed the base branch from main to compare-public-inputs August 23, 2023 18:34
@github-actions
Copy link

Benchmark Results for modified programs 🚀

Command Mean [s] Min [s] Max [s] Relative
head big_factorial 4.434 ± 0.018 4.413 4.464 1.00
Command Mean [s] Min [s] Max [s] Relative
head big_fibonacci 3.845 ± 0.011 3.832 3.862 1.00
Command Mean [s] Min [s] Max [s] Relative
head blake2s_integration_benchmark 10.793 ± 0.031 10.762 10.876 1.00
Command Mean [s] Min [s] Max [s] Relative
head compare_arrays_200000 3.898 ± 0.016 3.886 3.942 1.00
Command Mean [s] Min [s] Max [s] Relative
head dict_integration_benchmark 2.534 ± 0.008 2.521 2.547 1.00
Command Mean [s] Min [s] Max [s] Relative
head field_arithmetic_get_square_benchmark 1.883 ± 0.019 1.874 1.937 1.00
Command Mean [s] Min [s] Max [s] Relative
head integration_builtins 10.668 ± 0.020 10.641 10.712 1.00
Command Mean [s] Min [s] Max [s] Relative
head keccak_integration_benchmark 10.959 ± 0.017 10.935 10.992 1.00
Command Mean [s] Min [s] Max [s] Relative
head linear_search 3.882 ± 0.009 3.868 3.892 1.00
Command Mean [s] Min [s] Max [s] Relative
head math_cmp_and_pow_integration_benchmark 2.789 ± 0.004 2.780 2.797 1.00
Command Mean [s] Min [s] Max [s] Relative
head math_integration_benchmark 2.492 ± 0.005 2.483 2.497 1.00
Command Mean [s] Min [s] Max [s] Relative
head memory_integration_benchmark 2.214 ± 0.007 2.205 2.231 1.00
Command Mean [s] Min [s] Max [s] Relative
head operations_with_data_structures_benchmarks 2.447 ± 0.009 2.437 2.470 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
head pedersen 882.8 ± 1.6 881.3 886.2 1.00
Command Mean [s] Min [s] Max [s] Relative
head poseidon_integration_benchmark 1.570 ± 0.002 1.567 1.574 1.00
Command Mean [s] Min [s] Max [s] Relative
head secp_integration_benchmark 3.037 ± 0.009 3.029 3.060 1.00
Command Mean [s] Min [s] Max [s] Relative
head set_integration_benchmark 1.504 ± 0.002 1.501 1.506 1.00
Command Mean [s] Min [s] Max [s] Relative
head uint256_integration_benchmark 6.929 ± 0.027 6.903 6.976 1.00

This reverts commit 532dfca.
@codecov
Copy link

codecov bot commented Aug 23, 2023

Codecov Report

Merging #1394 (abd7895) into compare-public-inputs (a2736f7) will not change coverage.
The diff coverage is n/a.

@@                  Coverage Diff                   @@
##           compare-public-inputs    #1394   +/-   ##
======================================================
  Coverage                  97.51%   97.51%           
======================================================
  Files                         94       94           
  Lines                      38502    38502           
======================================================
  Hits                       37545    37545           
  Misses                       957      957           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions
Copy link

Benchmark Results for unmodified programs 🚀

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 5.349 ± 0.060 5.278 5.442 1.01 ± 0.01
head big_factorial 5.312 ± 0.039 5.247 5.366 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 4.780 ± 0.087 4.615 4.908 1.00
head big_fibonacci 4.936 ± 0.046 4.883 5.038 1.03 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 13.782 ± 0.225 13.561 14.275 1.02 ± 0.02
head blake2s_integration_benchmark 13.506 ± 0.196 13.258 13.952 1.00
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 4.737 ± 0.059 4.655 4.843 1.00
head compare_arrays_200000 4.773 ± 0.055 4.699 4.906 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 3.115 ± 0.055 3.057 3.230 1.01 ± 0.03
head dict_integration_benchmark 3.098 ± 0.059 3.047 3.250 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 2.173 ± 0.030 2.136 2.221 1.01 ± 0.02
head field_arithmetic_get_square_benchmark 2.157 ± 0.038 2.114 2.230 1.00
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 13.121 ± 0.621 12.645 14.691 1.02 ± 0.05
head integration_builtins 12.843 ± 0.079 12.736 13.009 1.00
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 13.572 ± 0.631 13.205 15.197 1.01 ± 0.06
head keccak_integration_benchmark 13.456 ± 0.553 13.152 14.994 1.00
Command Mean [s] Min [s] Max [s] Relative
base linear_search 4.645 ± 0.151 4.538 5.048 1.00 ± 0.04
head linear_search 4.625 ± 0.064 4.559 4.776 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 3.260 ± 0.037 3.216 3.310 1.00
head math_cmp_and_pow_integration_benchmark 3.261 ± 0.049 3.182 3.347 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 2.945 ± 0.043 2.894 3.021 1.00
head math_integration_benchmark 2.963 ± 0.047 2.898 3.046 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 2.645 ± 0.030 2.589 2.694 1.00
head memory_integration_benchmark 2.670 ± 0.071 2.596 2.843 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 2.845 ± 0.070 2.771 3.024 1.01 ± 0.03
head operations_with_data_structures_benchmarks 2.816 ± 0.022 2.777 2.850 1.00
Command Mean [s] Min [s] Max [s] Relative
base pedersen 1.064 ± 0.017 1.044 1.095 1.01 ± 0.03
head pedersen 1.059 ± 0.021 1.033 1.106 1.00
Command Mean [s] Min [s] Max [s] Relative
base poseidon_integration_benchmark 1.822 ± 0.024 1.794 1.864 1.00
head poseidon_integration_benchmark 1.830 ± 0.031 1.765 1.865 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 3.704 ± 0.097 3.637 3.960 1.01 ± 0.03
head secp_integration_benchmark 3.663 ± 0.049 3.558 3.712 1.00
Command Mean [s] Min [s] Max [s] Relative
base set_integration_benchmark 2.067 ± 0.033 2.026 2.132 1.00
head set_integration_benchmark 2.134 ± 0.062 2.067 2.239 1.03 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 8.074 ± 0.125 7.933 8.301 1.00
head uint256_integration_benchmark 8.137 ± 0.191 7.891 8.501 1.01 ± 0.03

@fmoletta fmoletta marked this pull request as ready for review August 23, 2023 20:58
@fmoletta fmoletta merged commit 5b8568f into compare-public-inputs Aug 28, 2023
@fmoletta fmoletta deleted the compare-inputs-in-ci branch August 28, 2023 19:48
github-merge-queue bot pushed a commit that referenced this pull request Aug 29, 2023
…s built (#1391)

* Remove cairo1 & cairo2 folders in `clean`  makefile target

* Make public inputs similar to python output

* Rename field

* Fix finalize method

* Fix get_public_memory logic (minus error handling)

* Add error handling

* Add comparison script

* Fix tests

* Add air_public_input argument to comparison script

* Add air public input comparison

* Fix finalize segments of output builtin

* Fix

* Add makefile targets

* Add new targets to phony

* Clean public input files

* Fix compilation in makefile

* Add changelog

* fmt

* Rename makefile targets

* Compare air_public_inputs in CI (#1394)

* Fix typo

* generate and compare air_public inputs in ci

* fix

* Fix syntax

* fix

* fix

* Revert "Fix typo"

This reverts commit 532dfca.

* Fix workflow

* Add .air_public_input to .gitignore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pipelines This PR/issue is exclusively about improving our CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants