VIP: vyper-json

[alcuadrado]

Simple Summary

Vyper should provide an interface for non-python-based tools to use.

Abstract

This VIP proposes the addition of another CLI tool: vyper-json. This tool will compile contracts, generating a structured JSON output containing everything needed to integrate Vyper, including errors.

Motivation

See #1492

Specification

This section describes the input and output interfaces of vyper-json.

Input description

vyper-json doesn't need any special input apart from a list of files to compile.

An example execution could be vyper-json path/to/contract1.vy path/to/other.vy

Output description

vyper-json outputs a single JSON object with one key of type PATH per input
file, associated with an object of type OUTPUT.

PATH values should be absolute paths. If symlinks are present, they should not
be resolved, keeping the path closer to the user's input.

OUTPUT values are objects with the following properties:

• files: An array of PATH values containing all the source files used to compile this contract. This is useful to implement things like cache, watchers, etc.
• errors: An array of ERROR objects. One per each fatal compilation error. This array must be empty if the compilation succeeded.
• warnings: An array of ERROR objects. One per each compilation warning. This array must be empty if no warning is emitted.
• output: An object containing bytecode, bytecode_runtime, abi, source_map, method_identifiers as returned by vyper when format combined_json is used. This value should be null if the compilation failed.

ERROR objects must contain the following properties:

• file: A PATH value pointing to the file where the warning or error originated.
• line: The error/warning's line number.
• column: The error/warning's line column.
• message: An explanation of the warning/error.

Example

Executing vyper-json path/to/contract1.vy path/to/other.vy

should output something like:

{
  "/absolute/path/to/contract1.vy": {
    "files": ["/absolute/path/to/contract1.vy", "/absolute/path/to/interface.vy"],
    "errors": [
      {
        "file": "/absolute/path/to/contract1.vy",
        "line": 1,
        "column": 20,
        "message": "A fatal error"
      }
    ],
    "warnings": [],
    "output": null
  },
  "/absolute/path/to/other.vy": {
    "files": ["/absolute/path/to/other.vy", "/path/path/to/ERC20.vy"],
    "errors": [],
    "warnings": [
      {
        "file": "/absolute/path/to/other.vy",
        "line": 2,
        "column": 30,
        "message": "A warning"
      }
    ],
    "output": { /* same as combined_json */ }
  }
}

Backwards Compatibility

This VIP proposes the introduction of a new command-line tool, which should have no impact on the existing one.

Dependencies

This VIP has no dependencies.

Copyright

Copyright and related rights waived via CC0

[fubuloubu]

I like it, did good work specifying this. IIRC solc's "json mode" requires the input to be a JSON string too. This would make it very easy to integrate with JavaScript tooling and web API endpoints.

[alcuadrado]

Yes, solidity's json mode expects a json containing everything it needs as input.

The problem with this approach is that in order to generate that json you need to parse the sources to get their imports and resolve them yourself. That's very problematic. Parsing correctly is not an easy task, and not having a standard imports resolution mechanism can lead to fragmentation in the tooling ecosystem.

[alcuadrado]

Some more info about Solidity's json input.

This last few days I've been working with the Etherscan's contract verification API and UI, and found that they now support using the standard json input. If it were supported in the API, that would solve lots of problems, as it contains all the info needed to reproduce a build. The key advantage is that it's self-contained in a single file.

My proposal for vyper-json doesn't have this property.

I still think that making imports resolution a responsibility of 3rd party tools is very problematic. Maybe have an official tool or vyper-json working mode that generates a self-contained input would solve both problems.

[fubuloubu]

Hmm, yeah maybe if you supply all the sources via JSON input, you can have the compiler reference via those strings instead of loading file I/O. Handle it in the script

[alcuadrado]

Maybe something like vyper-json --prepare-input file.vy | vyper-json?

The first call will take a file, and outputs a standarized json input with everything needed for compiling. That same json could be used for things like verifying the contract.

[fubuloubu]

Sure, as a shortcut tool. I was saying that vyper-json could work like so:

{
  "files": {
    "path/to/file1.vy": {
      "source": "# source code for file1..."
    },
    "path/to/file2.vy": {
      "source": "from path.to.file1 import interface..."
    }
  }
}

Internal to that script it would resolve the interfaces to the correct things.

[alcuadrado]

Sure, as a shortcut tool.

Yes, sure. It should return the errors (e.g. parsing errors when extracting the imports) in the same format that vyper-json.

[iamdefinitelyahuman]

This functionality would be very useful to me, if nobody else is already working on it I'd be happy to start right away.

To minimize friction when integrating Vyper with existing tooling, I think it would be beneficial to format the output similarly to Solidity's standard JSON output. It would look something like this:

  {
    // Optional: not present if no errors/warnings were encountered
    "errors": [
      {
        "sourceLocation": {
          "file": "path/to/contract1.vy",
          "start": 1,
          "end": 20
        },
        "type": "TypeError",
        "severity": "error", // ("error" or "warning")
        "message": "A fatal error or warning"
      }
    ],
    "sources": {
      "path/to/contract1.vy": {
        "ast": {}
      }
    },
    "contracts": {
      "path/to/contract1.vy": {
        "ContractName": {
          "abi": [],
          "evm": {
            "bytecode": {
              "object": "",
              "opcodes": "",
              "sourceMap": ""
            },
            "deployedBytecode": { 
              "object": "",
              "opcodes": "",
              "sourceMap": ""
            },
            "methodIdentifiers": {}
          }
        }
      }
    }
  }

I'm also happy to follow the previously discussed format. However the above approach will make my life easier down the line, and likely that of others looking to integrate Vyper where Solidity is already in use. Any thoughts?

[alcuadrado]

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

[fubuloubu]

There are a number of things that should/could be standardized between the two compilers, and I think this would be good first steps towards making that happen.

[iamdefinitelyahuman]

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

Indeed, some of the objects will need to be handled in different ways depending on if they're coming from vyper or solc. But not all of them. A common format would allow a script that deploys bytecode, for example, to handle vyper or solc-based contracts via a single method. More complex use cases will need to differentiate, but they can still start from the same place and diverge on a per-object basis as needed, rather than starting from two places and sometimes converging.

I do think you raise a valid concern - we should not assume that the program processing the json is the same that generated it, so there needs to be an easy way to look at the data and know what to expect within the individual objects. What if we include a field such as "compiler": "vyper-v0.1.0.-beta.11"? That way we still have simplified integration, but can prevent the headaches caused by having to determine the compiler type through looking at how the source map is formatted.

[fubuloubu]

I believe compiler is a field generated by the output json anyways (in Solidity's output)

[charles-cooper]

My worry about using that format is that there are many things that don't really match between solc and vyper outputs. For example, ast and source maps have completely different formats.

I don't believe standardization is within the scope of the VIP process but should be an EIP.

[fubuloubu]

Well yes, totally, I'm just saying starting down this road will eventually create the ability to standardize compiler interfaces for tools that use it, which would be best done in an EIP/ERC.

[jacqueswww]

Note from meeting: make vyper-json as compliant to solc as possible.

[charles-cooper]

for reference: https://solidity.readthedocs.io/en/v0.5.10/using-the-compiler.html#input-description

[iamdefinitelyahuman]

Will start work on this shortly