matrix foundry tests and snapshots #10022

RensR · 2023-08-01T15:05:49Z

This PR aims to improve the developer experience for developers writing Solidity through the following changes

Matrix testing for Foundry tests
- Each product (llo-feeds, vrf, ...) runs their Foundry tests in parallel
- Clear signalling on which product would break CI
- Currently it still runs all profiles regardless if a given product folder has changes. This is partly due to the Solidity folder not being properly cleaned up yet and many products are not contained to their designated folder.
- Also includes matrix based gas snapshot checks (see Per product gas snapshots section below)
Moved the Makefile for all Solidity based logic to /contracts so developers can run Foundry tests and the Make commands from the same location.
Per product gas snapshots
- Each product has its own gas snapshot, meaning updating the snapshot doesn't have to compile every contract
- Due to Foundry's caching limitation on compiling across Solidity versions, updating the gas snapshot always took significantly longer than just running tests for any given product.
- Based on the FOUNDRY_PROFILE env var and a new Make command make snapshot
- Since devs usually have the FOUNDRY_PROFILE already set during development to run forge test they can now run make snapshot to update only the snapshot for the product they're currently working on
Add product specific wrapper generation Make commands
- Uses the same FOUNDRY_PROFILE to indicate which wrappers are generated
- Also works for non-Foundry products
- Example code make wrappers with env var FOUNDRY_PROFILE=llo would compile all llo contracts and then create gethwrappers just for llo
- Is based on the split native_solc_compile_all_XXX compile scripts and core/gethwrappers/$(FOUNDRY_PROFILE)/go_generate.go generation files.
  - Not all products have a product specific go_generate yet, if you want to use this functionality, simply extract the lines that apply to your product into a properly named file to add support
Split generic Solidity CI and Hardhat CI
- Add ignore pattern to Hardhat CI to ignore product folders that don't use Hardhat
- This means that Hardhat CI can be skipped if changes are contained to those folders, speeding up CI very significantly as the new split Foundry CI takes only a single minute
  - You'll never have to wait for a ubuntu20.04-4cores-16GB machine to be available to run HH jobs

github-actions · 2023-08-01T15:06:18Z

I see that you haven't updated any README files. Would it make sense to do so?

github-actions · 2023-08-03T15:25:44Z

Go solidity wrappers are out-of-date, regenerate them via the make wrappers-all command

make running HH CI more restrictive

RensR · 2023-08-04T13:50:34Z

contracts/GNUmakefile

+# ALL_FOUNDRY_PRODUCTS contains a list of all products that have a foundry
+# profile defined. Adding a product to this list will make it available for
+# snapshotting.
+ALL_FOUNDRY_PRODUCTS = vrf automation llo functions automation-dev shared


automation-dev is required because automation has Foundry code in the dev folder. This should be moved into the automation folder at some point.

shared is already here even though there are no Foundry tests yet, this will soon change so I already added it

.github/workflows/solidity-foundry.yml

RensR · 2023-08-04T13:52:11Z

.github/workflows/solidity-hardhat.yml

+        id: changes
+        with:
+          filters: |
+            src:


This excludes quite a few files vs the old filter. Hopefully this results in running HH less often while never missing a run when needed

RensR · 2023-08-04T13:52:44Z

contracts/src/v0.8/shared/interfaces/ITypeAndVersion.sol

@@ -2,5 +2,5 @@
 pragma solidity ^0.8.0;

 interface ITypeAndVersion {
-  function typeAndVersion() external pure virtual returns (string memory);


fixing small mistake from previous PR

chainchad

CI looks good, just a few small suggestions.

.github/workflows/solidity-foundry.yml

.github/workflows/solidity-hardhat.yml

# Conflicts: # .github/workflows/solidity.yml # GNUmakefile # contracts/gas-snapshots/llo.gas-snapshot

contracts/GNUmakefile

matYang · 2023-08-14T14:22:54Z

contracts/GNUmakefile

+.PHONY: wrappers
+wrappers: pnpmdep abigen ## Recompiles solidity contracts and their go wrappers.
+	./scripts/native_solc_compile_all_$(FOUNDRY_PROFILE)
+	go generate ../core/gethwrappers/$(FOUNDRY_PROFILE)


this makes sense, but existing wrappers are placed under /core/gethwrappers/generated folder today, could this cause issues for go imports?

This only works for projects that adopt the new location. If they use the old way, they have to run make wrappers-all, just like now, and their files are unchanged. Some projects have already moved their wrapper generation code to subfolders, this command if for those teams.

cl-sonarqube-production · 2023-08-14T14:49:25Z

SonarQube Quality Gate

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

RensR added 3 commits August 3, 2023 17:00

matrix foundry tests and snapshots

6272aef

make snapshot great again

73a951e

add product based make wrapper commands

6722e11

RensR force-pushed the matrix-foundry-tests-&-snapshots branch from 1d9a6c3 to 6722e11 Compare August 3, 2023 15:00

update snapshots and reduce CI scope

aa7f705

RensR force-pushed the matrix-foundry-tests-&-snapshots branch from 49dfd88 to aa7f705 Compare August 3, 2023 15:13

RensR force-pushed the matrix-foundry-tests-&-snapshots branch from 0520558 to d6055da Compare August 3, 2023 15:35

use new wrapper gen in CI

669638e

RensR force-pushed the matrix-foundry-tests-&-snapshots branch from d6055da to 669638e Compare August 3, 2023 15:36

RensR added 5 commits August 3, 2023 18:22

split hardhat CI and generic Solidity CI

e034c06

make running HH CI more restrictive

Merge branch 'develop' into matrix-foundry-tests-&-snapshots

730fa1f

Merge branch 'develop' into matrix-foundry-tests-&-snapshots

830f483

Merge branch 'develop' into matrix-foundry-tests-&-snapshots

671ebbe

add shared Foundry profile & fix typeAndVersion

cccf49a

RensR marked this pull request as ready for review August 4, 2023 10:00

RensR requested review from chainchad, javuto, jkongie, jmank88, samsondav, se3000 and connorwstein as code owners August 4, 2023 10:00

RensR commented Aug 4, 2023

View reviewed changes

.github/workflows/solidity-foundry.yml Show resolved Hide resolved

RensR commented Aug 4, 2023

View reviewed changes

chainchad reviewed Aug 4, 2023

View reviewed changes

.github/workflows/solidity-foundry.yml Show resolved Hide resolved

.github/workflows/solidity-hardhat.yml Show resolved Hide resolved

.github/workflows/solidity-hardhat.yml Show resolved Hide resolved

.github/workflows/solidity-hardhat.yml Show resolved Hide resolved

clean up needs trigger and add comment

57f5472

RensR added 6 commits August 4, 2023 17:47

add foundry.toml CI trigger

3aa0b5b

Merge branch 'develop' into matrix-foundry-tests-&-snapshots

6fccdaf

# Conflicts: # .github/workflows/solidity.yml # GNUmakefile # contracts/gas-snapshots/llo.gas-snapshot

getwrapper generation in products folders

779866e

rename llo -> llo-feeds

d51f266

make snapshot-all

0a10fa6

foundryup && update snapshots

bb71bfe

matYang reviewed Aug 14, 2023

View reviewed changes

contracts/GNUmakefile Outdated Show resolved Hide resolved

matYang reviewed Aug 14, 2023

View reviewed changes

contracts/GNUmakefile Outdated Show resolved Hide resolved

matYang reviewed Aug 14, 2023

View reviewed changes

fix naming in comments

e471425

matYang approved these changes Aug 14, 2023

View reviewed changes

RensR added this pull request to the merge queue Aug 14, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 14, 2023

RensR added this pull request to the merge queue Aug 14, 2023

github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Aug 14, 2023

RensR added this pull request to the merge queue Aug 14, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 14, 2023

matYang added this pull request to the merge queue Aug 15, 2023

Merged via the queue into develop with commit 67b1f2d Aug 15, 2023
103 checks passed

matYang deleted the matrix-foundry-tests-&-snapshots branch August 15, 2023 04:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

matrix foundry tests and snapshots #10022

matrix foundry tests and snapshots #10022

RensR commented Aug 1, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023

github-actions bot commented Aug 3, 2023

RensR Aug 4, 2023

RensR Aug 4, 2023

RensR Aug 4, 2023

chainchad left a comment

matYang Aug 14, 2023

RensR Aug 14, 2023

cl-sonarqube-production bot commented Aug 14, 2023

matrix foundry tests and snapshots #10022

matrix foundry tests and snapshots #10022

Conversation

RensR commented Aug 1, 2023 • edited Loading

github-actions bot commented Aug 1, 2023

github-actions bot commented Aug 3, 2023

RensR Aug 4, 2023

Choose a reason for hiding this comment

RensR Aug 4, 2023

Choose a reason for hiding this comment

RensR Aug 4, 2023

Choose a reason for hiding this comment

chainchad left a comment

Choose a reason for hiding this comment

matYang Aug 14, 2023

Choose a reason for hiding this comment

RensR Aug 14, 2023

Choose a reason for hiding this comment

cl-sonarqube-production bot commented Aug 14, 2023

RensR commented Aug 1, 2023 •

edited

Loading