[WIP] Add tests for script invocation

[davesque]

What I did

Added a test suite to ensure proper functioning of bin scripts.

How I did it

Added pytest-console-scripts plugin to testing requirements and bumped required pytest version to take advantage of built-in temp file fixtures. Added test suite for scripts.

How to verify it

Run tests.

Cute Animal Picture

[fubuloubu]

@jacqueswww how fragile would it be to test against raw bytecode for these simple programs? It shouldn't have to be updated too often due to language semantics changes, right?

[pipermerriam]

Since these tests are against the raw compiled output I'm curious to explore ways to make them less fragile. It seems like lots of things could change the resulting bytecode output making these tests fragile (but still valuable). Here are some thoughts.

1. Instead of testing the raw output, test the resulting contract functionality. Parse the output -> deploy the contract -> call some functions -> test the returned values.
2. Property based testing.

• raw output should contain the 0x61... preamble
• raw output should contain the PUSH <42-as-literal> sequence of instructions.
• raw output should be hex string of at least length XXX

3. Hamming distance between expected output and actual output is within X, to allow for small changes to resulting bytecode.

[pipermerriam]

Using an itertools.count for this type of thing can be a bit cleaner in cases like this where you just need blind increments. I think it allows you to remove the need for nonlocal and in theory you can inline the incrementing into the string formating via f'{tmp{next(name_counter)}.vy' if you're feeling brave.

[davesque]

Couldn't having a more flexible test be just as brittle but in a way that's harder to detect? For example, imagine we're searching for the preamble and push op. Then, someone breaks the compiler by causing the push op to appear in the constructor instead of the method. We wouldn't know anything was wrong (actually, I guess we probably would because other tests would almost certainly fail but you can still see what I mean). On the other hand, if you are making a change that has that effect, you expect to see tests break. Looking for a literal value gives the highest likelihood that the tests will fail in an expected way.

[pipermerriam]

Couldn't having a more flexible test be just as brittle

Yes, though I'd argue that these are integration tests or something like that, and like you said, we focus on catching functionality errors at a lower level, and design this suite to ensure the CLI compilation produces the expected output. I recognize the impreciseness of that definition and the somewhat circular definition. Only trying to brainstorm ways to get similar guarantees from the tests while reducing likelihood that they end up being brittle.

[jacqueswww]

Just bombing in to say this is PR leet congrats :-)

P.S. will check this PR out tomorrow.

[davesque]

@jacqueswww @pipermerriam Just as a reminder, I see this PR as being very far from complete! It was really just the first couple of steps. :)

[davesque]

@jacqueswww Okay, haha. I just looked at the PR number and now I get what you meant :).

[fubuloubu]

@davesque any updates on this one?

[davesque]

@fubuloubu Not currently. I can potentially get back on this after doing the eth-abi strategies and juggle it with the AST class work.

[fubuloubu]

Awesome! Would be helpful I think, we've had a few strange issues since the script is largely untested.

[davesque]

@pipermerriam I'm looking over this discussion some time later now and, in retrospect, I completely agree with the points you were making earlier. We could probably find ways to make things a bit more robust.

[fubuloubu]

@davesque The original premise of this PR is now no longer necessary due to #1576, although I am sure some of the work can be reused to the new configuration as the compiler endpoints in vyper/cli are undertested (but should be easier to test now!)