feat: add support for modules with variables #3707

charles-cooper · 2023-12-22T01:58:51Z

What I did

add support for modules with variables. modules are instantiated separately from import statements, decoupling code and storage layout concerns.

How I did it

change the allocator to handle modules as variables
add analysis support for touched locations
change the calling convention to include pointers to storage/immutables/whatever

note that this feature probably requires dropping support for storage layout overrides (at least in its current form)

How to verify it

still incomplete. need to make sure support for immutables is correct, and add tests.

unit tests for the analysis
various combinations of importing modules with storage and/or immutables
tests for the storage allocator
handling of __init__() functions

Commit message

Commit message for the final, squashed PR. (Optional, but reviewers will appreciate it! Please see our commit message style guide for what we would ideally like to see in a commit message.)

Description for the changelog

Cute Animal Picture

kill ModuleInfo, can just use VarInfo now

…tribute

…rType storage_size_in_words -> storage_slots_required immutable_section_bytes -> immutable_bytes_required this establishes a convention to find the size of a type: `typ.{location}_{slots_or_bytes}_required` this maybe indicates that a better API will be `size_in(location: DataLocation)` but we will see how the design evolves

- rename module_ctx to compilation_target

vyper/codegen/self_call.py

+from vyper.codegen.core import (
+    _freshname,
+    data_location_to_addr_space,
+    eval_once_check,
+    get_element_ptr,
+    make_setter,
+)


vyper/codegen/context.py

@@ -94,6 +92,24 @@
        # either the constructor, or called from the constructor
        self.is_ctor_context = is_ctor_context

+    def self_ptr(self, location):


vyper/codegen/self_call.py

DanielSchiavini · 2023-12-22T13:30:06Z

vyper/codegen/core.py

+    if s == DataLocation.STORAGE:
+        return STORAGE
+    if s == DataLocation.MEMORY:
+        return MEMORY
+    if s == DataLocation.CODE:
+        return IMMUTABLES
+
+    raise CompilerPanic("unreachable")  # pragma: nocover


Suggested change

if s == DataLocation.STORAGE:

return STORAGE

if s == DataLocation.MEMORY:

return MEMORY

if s == DataLocation.CODE:

return IMMUTABLES

raise CompilerPanic("unreachable") # pragma: nocover

return {

DataLocation.STORAGE: STORAGE,

DataLocation.MEMORY: MEMORY,

DataLocation.CODE: IMMUTABLES,

}[s]

DanielSchiavini · 2023-12-22T13:32:42Z

vyper/codegen/core.py

+    for i in range(index):
+        ofst += module_t.variables[attrs[i]].typ.storage_slots_required


Suggested change

for i in range(index):

ofst += module_t.variables[attrs[i]].typ.storage_slots_required

offset_from_parent = sum(

module_t.variables[var].typ.storage_slots_required

for var in attrs[:index]

)

DanielSchiavini · 2023-12-22T13:38:14Z

vyper/codegen/self_call.py

+    module_t = func_t.ast_def._parent._metadata["type"]
+    if module_t == context.compilation_target:
+        # we don't need to pass a pointer
+        return ret


Suggested change

return ret

return []

DanielSchiavini · 2023-12-22T14:02:35Z

vyper/semantics/types/function.py

+        for r in self._variable_reads:
+            if r._metadata["variable_access"].location == location:
+                return True
+        for w in self._variable_writes:
+            if w._metadata["variable_write"].location == location:
+                return True
+        return False


Suggested change

for r in self._variable_reads:

if r._metadata["variable_access"].location == location:

return True

for w in self._variable_writes:

if w._metadata["variable_write"].location == location:

return True

return False

from itertools import concat

return any(

v._metadata["variable_access"].location == location

for v in concat(self._variable_reads, self._variable_writes)

)

DanielSchiavini · 2023-12-22T14:04:55Z

vyper/semantics/types/function.py

+        ret = []
+        possible_locations = [DataLocation.STORAGE, DataLocation.CODE]
+        for location in possible_locations:
+            if self.touches_location(location):
+                ret.append(location)
+        return ret


Suggested change

ret = []

possible_locations = [DataLocation.STORAGE, DataLocation.CODE]

for location in possible_locations:

if self.touches_location(location):

ret.append(location)

return ret

possible_locations = (DataLocation.STORAGE, DataLocation.CODE)

return [

location for location in possible_locations

if self.touches_location(location)

]

per title, replace `enum` with `flag` as it more closely models https://docs.python.org/3/library/enum.html#enum.IntFlag than regular enums. allow `enum` for now (for backwards compatibility) but convert to `flag` internally and issue a warning

rename `assert_tx_failed` to `tx_failed` and change it into a context manager which has a similar API to `pytest.raises()`. --------- Co-authored-by: Charles Cooper <cooper.charles.m@gmail.com>

- allow range where both start and end arguments are variables, so long as a bound is supplied - ban range expressions of the form `range(x, x + N)` since the new form is cleaner and supersedes it. - also do a bit of refactoring of the codegen for range --------- Co-authored-by: Charles Cooper <cooper.charles.m@gmail.com>

* feat: improve panics in IR generation this QOL commit improves on 9165926 by passing through the `__traceback__` field when the exception is modified (instead of using `__cause__` - cf. PEP-3134 regarding the difference) and improves error messages when an IRnode is not returned properly. using `__traceback__` generally results in a better experience because the immediate cause of the exception is displayed when running `vyper -v` instead of needing to scroll up through the exception chain (if the exception chain is reproduced correctly at all in the first place). --------- Co-authored-by: Harry Kalogirou <harkal@nlogn.eu>

vyper/codegen/core.py

    StructT,
    TupleT,
    _BytestringT,
 )
 from vyper.semantics.types.shortcuts import BYTES32_T, INT256_T, UINT256_T
 from vyper.semantics.types.subscriptable import SArrayT
-from vyper.semantics.types.user import EnumT
+from vyper.semantics.types.user import FlagT


vyper/semantics/analysis/utils.py

@@ -248,13 +245,13 @@
        # comparisons, e.g. `x < y`

        # TODO fixme circular import
-        from vyper.semantics.types.user import EnumT
+        from vyper.semantics.types.user import FlagT


and wip on ModuleVarInfo

- split VarInfo positions API from get_position/set_position to get_storage_position/set_storage_position and ditto for immutables - only one of these is valid for regular VarInfos, but modules get allocated in both storage and immutables. - rewrite set_code_offsets to share allocator code - rename set_data_positions to allocate_module_variables

vyper/semantics/analysis/utils.py

vyper/semantics/analysis/base.py

+    def __post_init__(self):
+        super().__post_init__()
+        # hmm
+        from vyper.semantics.types.module import ModuleT


vyper/codegen/expr.py

@@ -36,6 +36,7 @@
    VyperException,
    tag_exceptions,
 )
+from vyper.semantics.analysis.base import VarInfo


vyper/semantics/analysis/utils.py

@@ -17,7 +17,7 @@
    ZeroDivisionException,
 )
 from vyper.semantics import types
-from vyper.semantics.analysis.base import ExprInfo, Modifiability, ModuleInfo, VarInfo
+from vyper.semantics.analysis.base import ExprInfo, Modifiability, ModuleVarInfo, VarInfo


vyper/semantics/analysis/utils.py

@@ -17,7 +17,7 @@
    ZeroDivisionException,
 )
 from vyper.semantics import types
-from vyper.semantics.analysis.base import ExprInfo, Modifiability, ModuleInfo, VarInfo
+from vyper.semantics.analysis.base import ExprInfo, Modifiability, ModuleVarInfo, VarInfo


charles-cooper · 2024-02-13T00:48:28Z

superseded by #3729

charles-cooper added 15 commits December 20, 2023 18:40

wip - module variables

a59d575

kill ModuleInfo, can just use VarInfo now

add size_in_bytes to module

e353a15

wip - storage allocator

33dffaf

remove ImportedVariable thing

654256b

wip - add get_element_ptr for module, fix some logic in Expr.parse_At…

bcc03d6

…tribute

call set_data_positions recursively

e9b867a

add a sanity check

037d5a6

add a comment

86c299a

add Context.self_ptr helper

9e689b9

- rename module_ctx to compilation_target

add a note

8becda2

Merge branch 'master' into feat/module_variables

a0d0bd1

improve Context.self_ptr

9ac072b

add function variable read/writes analysis

4e102bf

calculate pointer things

9f100d9

github-advanced-security bot found potential problems Dec 22, 2023

View reviewed changes

quash mypy

2d05699

github-advanced-security bot found potential problems Dec 22, 2023

View reviewed changes

vyper/codegen/self_call.py Fixed Show fixed Hide fixed

wip - handle immutables

1585bdc

DanielSchiavini reviewed Dec 22, 2023

View reviewed changes

AlbertoCentonze and others added 5 commits December 27, 2023 09:40

refactor: make assert_tx_failed a contextmanager (vyperlang#3706)

1824321

rename `assert_tx_failed` to `tx_failed` and change it into a context manager which has a similar API to `pytest.raises()`. --------- Co-authored-by: Charles Cooper <cooper.charles.m@gmail.com>

add special visibility for the __init__ function

977851a

github-advanced-security bot found potential problems Dec 27, 2023

View reviewed changes

charles-cooper added 4 commits December 28, 2023 12:49

remove unused MemoryOffset, CalldataOffset classes

8af611b

wip - allow init functions to be called from init func

c241e91

rename DataLocations.CODE to IMMUTABLES

3de7bb2

and wip on ModuleVarInfo

charles-cooper added 3 commits December 29, 2023 18:43

thread new offset through codegen

5e08300

mark storage layout override tests as xfail

1e393fa

Merge branch 'master' into feat/module_variables

6cf9ff0

github-advanced-security bot found potential problems Dec 30, 2023

View reviewed changes

charles-cooper added 2 commits January 2, 2024 12:33

Merge branch 'master' into feat/module_variables

7a82180

Merge branch 'master' into feat/module_variables

eff76e0

github-advanced-security bot found potential problems Jan 2, 2024

View reviewed changes

charles-cooper mentioned this pull request Jan 13, 2024

VIP: instantiable stateful modules #3723

Closed

charles-cooper closed this Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for modules with variables #3707

feat: add support for modules with variables #3707

charles-cooper commented Dec 22, 2023 •

edited

Loading

DanielSchiavini Dec 22, 2023

DanielSchiavini Dec 22, 2023

DanielSchiavini Dec 22, 2023

DanielSchiavini Dec 22, 2023

DanielSchiavini Dec 22, 2023

charles-cooper commented Feb 13, 2024

		for i in range(index):
		ofst += module_t.variables[attrs[i]].typ.storage_slots_required

-    for i in range(index):
-        ofst += module_t.variables[attrs[i]].typ.storage_slots_required
+    offset_from_parent = sum(
+        module_t.variables[var].typ.storage_slots_required
+        for var in attrs[:index]
+    )

feat: add support for modules with variables #3707

feat: add support for modules with variables #3707

Conversation

charles-cooper commented Dec 22, 2023 • edited Loading

What I did

How I did it

How to verify it

Commit message

Description for the changelog

Cute Animal Picture

DanielSchiavini Dec 22, 2023

Choose a reason for hiding this comment

DanielSchiavini Dec 22, 2023

Choose a reason for hiding this comment

DanielSchiavini Dec 22, 2023

Choose a reason for hiding this comment

DanielSchiavini Dec 22, 2023

Choose a reason for hiding this comment

DanielSchiavini Dec 22, 2023

Choose a reason for hiding this comment

charles-cooper commented Feb 13, 2024

charles-cooper commented Dec 22, 2023 •

edited

Loading