Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Approval testing for the Analyzer #387

Merged
merged 1 commit into from
May 5, 2021
Merged

Approval testing for the Analyzer #387

merged 1 commit into from
May 5, 2021

Conversation

g-r-a-n-t
Copy link
Member

@g-r-a-n-t g-r-a-n-t commented May 3, 2021 •
edited
Loading

What was wrong?

Testing around the Analyzer was not very thorough.

How was it fixed?

We now generate snapshots of Context using insta. The snapshots are formatted using codespan and the debug implementation of attributes. Here are a few things to note:

  • Nodes are labeled with the hash of their attributes. This makes it so that the set of changed nodes is clearly visible and lets us place the attributes debug further down, where it reads more easily.
  • Usage of hash maps and sets were removed in favor of b trees. This makes ordering consistent between snapshots.
  • The snapshot format is clear enough for someone to skim over and gain meaningful insight.
  • For now, I've copied the fixture directory into the analyzer, but when Initial implementation of differential fuzzing #385 is complete, we should move these tests into the testing crate and use the other fixture directory. TODO: write an issue for this

As an example, this is what would show up during insta review if you changed the location of numeric literals from value to memory:

insta-attributes

Notice that the same node is displayed twice. This is because the list of nodes used to generate the output is built by pushing nodes passed into context.add_ methods. These add methods are called twice in some situations, so we end up with duplicates. I think this should be fixed by adding update methods to Context, so we're not calling adds multiple times. TODO: write an issue for this

To-Do

  • OPTIONAL: Update Spec if applicable
  • Add entry to the release notes (may forgo for trivial changes)
  • Clean up commit history
  • Create the issues mentioned in this comment.

@g-r-a-n-t g-r-a-n-t changed the title Serialize context Approval testing for the Analyzer May 3, 2021
@g-r-a-n-t g-r-a-n-t force-pushed the serialize-context branch from 8b591f7 to 2be1191 Compare May 3, 2021 19:18
@codecov-commenter
Copy link

Codecov Report

Merging #387 (2be1191) into master (5c25397) will increase coverage by 0.14%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #387      +/-   ##
==========================================
+ Coverage   86.30%   86.45%   +0.14%     
==========================================
  Files          67       67              
  Lines        4032     4075      +43     
==========================================
+ Hits         3480     3523      +43     
  Misses        552      552
Impacted Files Coverage Δ
analyzer/src/namespace/events.rs 100.00% <ø> (ø)
parser/src/node.rs 76.47% <ø> (ø)
analyzer/src/context.rs 92.77% <100.00%> (+2.26%) ⬆️
analyzer/src/namespace/scopes.rs 95.93% <100.00%> (ø)
analyzer/src/namespace/types.rs 89.75% <100.00%> (ø)
common/src/diagnostics.rs 50.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c25397...2be1191. Read the comment docs.

g-r-a-n-t
g-r-a-n-t commented May 4, 2021
View reviewed changes
analyzer/Cargo.toml Outdated Show resolved Hide resolved
analyzer/src/context.rs Show resolved Hide resolved
common/src/diagnostics.rs Outdated Show resolved Hide resolved
@g-r-a-n-t g-r-a-n-t force-pushed the serialize-context branch from 2827ae2 to 6050914 Compare May 4, 2021 22:16
g-r-a-n-t
g-r-a-n-t commented May 4, 2021
View reviewed changes
analyzer/tests/fixtures Outdated Show resolved Hide resolved
@g-r-a-n-t g-r-a-n-t force-pushed the serialize-context branch 3 times, most recently from f90e902 to 525aa09 Compare May 4, 2021 23:57
@g-r-a-n-t g-r-a-n-t force-pushed the serialize-context branch from 525aa09 to c3b1f31 Compare May 4, 2021 23:59
@g-r-a-n-t g-r-a-n-t requested review from cburgdorf and sbillig May 5, 2021 00:34
cburgdorf
cburgdorf approved these changes May 5, 2021
View reviewed changes
Copy link
Collaborator

@cburgdorf cburgdorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, left a question to double check I'm not misunderstanding what this does 😅

analyzer/tests/analysis.rs
case::empty("features/empty.fe"),
case::events("features/events.fe"),
case::external_contract("features/external_contract.fe"),
case::for_loop_with_break("features/for_loop_with_break.fe"),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double check: We now test the state of the context for all these fixtures as an additional category of tests that we run. These tests do not actually test the return values of the called functions which we continue to do as before here. Is that right?

fe/compiler/tests/cases/features.rs

Lines 112 to 237 in c3b1f31

#[rstest(fixture_file, input, expected,
case("for_loop_with_static_array.fe", &[], uint_token(30)),
case("for_loop_with_break.fe", &[], uint_token(15)),
case("for_loop_with_continue.fe", &[], uint_token(17)),
case("while_loop_with_continue.fe", &[], uint_token(1)),
case("while_loop.fe", &[], uint_token(3)),
case("while_loop_with_break.fe", &[], uint_token(1)),
case("while_loop_with_break_2.fe", &[], uint_token(1)),
case("if_statement.fe", &[uint_token(6)], uint_token(1)),
case("if_statement.fe", &[uint_token(4)], uint_token(0)),
case("if_statement_2.fe", &[uint_token(6)], uint_token(1)),
case("if_statement_with_block_declaration.fe", &[], uint_token(1)),
case("ternary_expression.fe", &[uint_token(6)], uint_token(1)),
case("ternary_expression.fe", &[uint_token(4)], uint_token(0)),
case("call_statement_without_args.fe", &[], uint_token(100)),
case("call_statement_with_args.fe", &[], uint_token(100)),
case("call_statement_with_args_2.fe", &[], uint_token(100)),
case("return_bool_true.fe", &[], bool_token(true)),
case("return_bool_false.fe", &[], bool_token(false)),
case("return_bool_inverted.fe", &[bool_token(true)], bool_token(false)),
case("return_bool_inverted.fe", &[bool_token(false)], bool_token(true)),
case("return_u256_from_called_fn_with_args.fe", &[], uint_token(200)),
case("return_u256_from_called_fn.fe", &[], uint_token(42)),
case("return_u256.fe", &[], uint_token(42)),
case("return_i256.fe", &[], int_token(-3)),
case("return_identity_u256.fe", &[uint_token(42)], uint_token(42)),
case("return_identity_u128.fe", &[uint_token(42)], uint_token(42)),
case("return_identity_u64.fe", &[uint_token(42)], uint_token(42)),
case("return_identity_u32.fe", &[uint_token(42)], uint_token(42)),
case("return_identity_u16.fe", &[uint_token(42)], uint_token(42)),
case("return_identity_u8.fe", &[uint_token(42)], uint_token(42)),
case("return_u128_cast.fe", &[], uint_token(42)),
case("return_i128_cast.fe", &[], int_token(-3)),
case("return_msg_sig.fe", &[], bytes32_token("febb0f7e")),
// binary operators
case("return_addition_u256.fe", &[uint_token(42), uint_token(42)], uint_token(84)),
case("return_addition_i256.fe", &[int_token(-42), int_token(-42)], int_token(-84)),
case("return_addition_i256.fe", &[int_token(-42), int_token(42)], int_token(0)),
case("return_addition_u128.fe", &[uint_token(42), uint_token(42)], uint_token(84)),
case("return_subtraction_u256.fe", &[uint_token(42), uint_token(42)], uint_token(0)),
case("return_subtraction_i256.fe", &[int_token(-42), int_token(-42)], int_token(0)),
case("return_subtraction_i256.fe", &[int_token(-42), int_token(42)], int_token(-84)),
case("return_multiplication_u256.fe", &[uint_token(42), uint_token(42)], uint_token(1764)),
case("return_multiplication_i256.fe", &[int_token(-42), int_token(-42)], int_token(1764)),
case("return_multiplication_i256.fe", &[int_token(-42), int_token(42)], int_token(-1764)),
case("return_division_u256.fe", &[uint_token(42), uint_token(42)], uint_token(1)),
case("return_division_i256.fe", &[int_token(-42), int_token(-42)], int_token(1)),
case("return_division_i256.fe", &[int_token(-1), int_token(1)], int_token(-1)),
case("return_division_i256.fe", &[int_token(-42), int_token(42)], int_token(-1)),
case("return_pow_u256.fe", &[uint_token(2), uint_token(0)], uint_token(1)),
case("return_pow_u256.fe", &[uint_token(2), uint_token(4)], uint_token(16)),
case("return_pow_i256.fe", &[int_token(-2), uint_token(3)], int_token(-8)),
case("return_mod_u256.fe", &[uint_token(5), uint_token(2)], uint_token(1)),
case("return_mod_u256.fe", &[uint_token(5), uint_token(3)], uint_token(2)),
case("return_mod_u256.fe", &[uint_token(5), uint_token(5)], uint_token(0)),
case("return_mod_i256.fe", &[int_token(5), int_token(2)], int_token(1)),
case("return_mod_i256.fe", &[int_token(5), int_token(3)], int_token(2)),
case("return_mod_i256.fe", &[int_token(5), int_token(5)], int_token(0)),
case("return_bitwiseand_u256.fe", &[uint_token(12), uint_token(25)], uint_token(8)),
case("return_bitwiseand_u128.fe", &[uint_token(12), uint_token(25)], uint_token(8)),
case("return_bitwiseor_u256.fe", &[uint_token(12), uint_token(25)], uint_token(29)),
case("return_bitwisexor_u256.fe", &[uint_token(12), uint_token(25)], uint_token(21)),
case("return_bitwiseshl_u256.fe", &[uint_token(212), uint_token(0)], uint_token(212)),
case("return_bitwiseshl_u256.fe", &[uint_token(212), uint_token(1)], uint_token(424)),
case("return_bitwiseshr_u256.fe", &[uint_token(212), uint_token(0)], uint_token(212)),
case("return_bitwiseshr_u256.fe", &[uint_token(212), uint_token(1)], uint_token(106)),
case("return_bitwiseshr_i256.fe", &[int_token(212), uint_token(0)], int_token(212)),
case("return_bitwiseshr_i256.fe", &[int_token(212), uint_token(1)], int_token(106)),
// comparison operators
case("return_eq_u256.fe", &[uint_token(1), uint_token(1)], bool_token(true)),
case("return_eq_u256.fe", &[uint_token(1), uint_token(2)], bool_token(false)),
case("return_noteq_u256.fe", &[uint_token(1), uint_token(1)], bool_token(false)),
case("return_noteq_u256.fe", &[uint_token(1), uint_token(2)], bool_token(true)),
case("return_lt_u256.fe", &[uint_token(1), uint_token(2)], bool_token(true)),
case("return_lt_u256.fe", &[uint_token(1), uint_token(1)], bool_token(false)),
case("return_lt_u256.fe", &[uint_token(2), uint_token(1)], bool_token(false)),
case("return_lt_u128.fe", &[uint_token(1), uint_token(2)], bool_token(true)),
// lt_i256 with positive and negative numbers
case("return_lt_i256.fe", &[int_token(1), int_token(2)], bool_token(true)),
case("return_lt_i256.fe", &[int_token(1), int_token(1)], bool_token(false)),
case("return_lt_i256.fe", &[int_token(2), int_token(1)], bool_token(false)),
case("return_lt_i256.fe", &[int_token(-2), int_token(-1)], bool_token(true)),
case("return_lt_i256.fe", &[int_token(-1), int_token(-1)], bool_token(false)),
case("return_lt_i256.fe", &[int_token(-1), int_token(-2)], bool_token(false)),
case("return_lte_u256.fe", &[uint_token(1), uint_token(2)], bool_token(true)),
case("return_lte_u256.fe", &[uint_token(1), uint_token(1)], bool_token(true)),
// lte_i256 with positive and negative numbers
case("return_lte_u256.fe", &[uint_token(2), uint_token(1)], bool_token(false)),
case("return_lte_i256.fe", &[int_token(1), int_token(2)], bool_token(true)),
case("return_lte_i256.fe", &[int_token(1), int_token(1)], bool_token(true)),
case("return_lte_i256.fe", &[int_token(2), int_token(1)], bool_token(false)),
case("return_lte_i256.fe", &[int_token(-2), int_token(-1)], bool_token(true)),
case("return_lte_i256.fe", &[int_token(-1), int_token(-1)], bool_token(true)),
case("return_lte_i256.fe", &[int_token(-1), int_token(-2)], bool_token(false)),
case("return_gt_u256.fe", &[uint_token(2), uint_token(1)], bool_token(true)),
case("return_gt_u256.fe", &[uint_token(1), uint_token(1)], bool_token(false)),
case("return_gt_u256.fe", &[uint_token(1), uint_token(2)], bool_token(false)),
// gt_i256 with positive and negative numbers
case("return_gt_i256.fe", &[int_token(2), int_token(1)], bool_token(true)),
case("return_gt_i256.fe", &[int_token(1), int_token(1)], bool_token(false)),
case("return_gt_i256.fe", &[int_token(1), int_token(2)], bool_token(false)),
case("return_gt_i256.fe", &[int_token(-1), int_token(-2)], bool_token(true)),
case("return_gt_i256.fe", &[int_token(-1), int_token(-1)], bool_token(false)),
case("return_gt_i256.fe", &[int_token(-2), int_token(-1)], bool_token(false)),
case("return_gte_u256.fe", &[uint_token(2), uint_token(1)], bool_token(true)),
case("return_gte_u256.fe", &[uint_token(1), uint_token(1)], bool_token(true)),
case("return_gte_u256.fe", &[uint_token(1), uint_token(2)], bool_token(false)),
// gte_i256 with positive and negative numbers
case("return_gte_i256.fe", &[int_token(2), int_token(1)], bool_token(true)),
case("return_gte_i256.fe", &[int_token(1), int_token(1)], bool_token(true)),
case("return_gte_i256.fe", &[int_token(1), int_token(2)], bool_token(false)),
case("return_gte_i256.fe", &[int_token(-1), int_token(-2)], bool_token(true)),
case("return_gte_i256.fe", &[int_token(-1), int_token(-1)], bool_token(true)),
case("return_gte_i256.fe", &[int_token(-2), int_token(-1)], bool_token(false)),
// `and` bool operation
case("return_bool_op_and.fe", &[bool_token(true), bool_token(true)], bool_token(true)),
case("return_bool_op_and.fe", &[bool_token(true), bool_token(false)], bool_token(false)),
case("return_bool_op_and.fe", &[bool_token(false), bool_token(true)], bool_token(false)),
case("return_bool_op_and.fe", &[bool_token(false), bool_token(false)], bool_token(false)),
// `or` bool operation
case("return_bool_op_or.fe", &[bool_token(true), bool_token(true)], bool_token(true)),
case("return_bool_op_or.fe", &[bool_token(true), bool_token(false)], bool_token(true)),
case("return_bool_op_or.fe", &[bool_token(false), bool_token(true)], bool_token(true)),
case("return_bool_op_or.fe", &[bool_token(false), bool_token(false)], bool_token(false)),
)]
fn test_method_return(fixture_file: &str, input: &[ethabi::Token], expected: ethabi::Token) {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we are just testing the state of the context. Anytime we change something in the analyzer that results in a different context output for a given fixture, it will show up here.

There have been a number of times where I've been reworking things in the analyzer and ended up breaking some unrelated full-compilation fixture test. Resolving these types of issues is tedious, since it's not clear what changed in the analyzer output. With this, it will be obvious what changed.

@g-r-a-n-t g-r-a-n-t merged commit 77090b3 into ethereum:master May 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cburgdorf cburgdorf cburgdorf approved these changes

@sbillig sbillig Awaiting requested review from sbillig

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@g-r-a-n-t @codecov-commenter @cburgdorf