-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Implement malloc, free, and realloc for WASM #129
Comments
Famous last words |
hi,very nice |
I have some questions on these requirements. How much effort are we wanting to put into memory pool management to avoid ending up with a fragmented address space? (i.e. would we rather have a naive approach that will definitely be completed in time, vs. a more useful solution for the long run?) |
We are not adding intrinsic functions, we are implementing this entirely within C++ headers of EOSLIB. |
Will need to add a memory.h file to EOSLIB and allocate a static char buffer. Initial iteration will just have alloc and realloc grabbing the next unallocated region, free will be a no-op. Along with first iteration need to cerate a test to print out location of buffer as well as create a method to print out the location of a local variable in the method and recursively call itself to identify how the wasm C++ stack grows and if we will need to worry about it eventually overwriting our buffer. Also need to identify how big of an initial buffer we can allocate. |
It looks like our block of memory is being allocated early in the memory space (offset 12 is the start, for the create_script test I ran). The C++ stack seems to start somewhere around 10000 (printing the address of my local variable in my recursive function is offset 9884) and there is nothing preventing either of them from walking over the other. If I allocate 8K create_script runs fine, anything close to 9884 causes bad behavior. Similarly if I have my recursive function recurse enough times to overwrite past 8K, no system errors are reported, just system instability. |
Adding some more tests and will create a pull request tomorrow am. Will then go on to 2nd iteration and add 64K pages (up to 1M) and test to identify how fast page adding is. |
s2wasm help says:
Our cmake uses -s 1024 Dan |
Did some further tests with allocate stack setting and added some more tests for memory.hpp. Submitting pull request. |
Working on peer review changes. Would like @bytemaster 's input on if we want to put test contracts (those that are not examples like exchange, and are only for "unit testing" of eoslib or like infinity that tests that we eventually timeout) I am suggesting that these get put under tests/slow_tests/ or tests/api_tests/ so that contracts doesn't get cluttered with examples that should not be followed. |
@bytemaster I was going to move on to adding extra page memory when the buffer is exhausted, but will switch over to writing tests for #215 since that is marked as a priority, unless I hear otherwise from you. |
This has a preliminary implementation, with 8K of memory and not freeing memory. It is correctly marked for EOS Dawn 2.0. |
Working on understanding the memory allocation in WASM (MemoryInstance), expanding the memory space and how to ensure that memory is only accessed by malloc and realloc. And a lot of testing along with it. |
@bytemaster Is the plumbing currently in place to create the new 64K page and make it available for use? I know that pieces are in there like Runtime::growMemory, but I don't see anything that provides what is needed (lookup of ModuleInstance) other than figuring out what C++ code would translate to the WASM grow_memory opcode. I will move onto managing the memory and circle back once you can comment on this. |
Refactoring the memory handling to maintain multiple buffers, and debugging. |
Didn't make much progress on this because of bug #481. Will start back on this in the morning. |
Should be able to commit some tests for my refactored code. Until I get answers to my earlier post to @bytemaster I have some hack code to allow the code to be tested (I just increase the size of the initial heap memory and divvy it out as if it is new page memory), and will remove it when I figure out about the page memory plumbing. |
While I wait for pull request comments and answers from @bytemaster about adding the 64K memory pages, I will move on to implementing a simple approach to free till we can discuss this more as well. |
Simple approach to free is trying to maintain the list of free memory, without having to have logic to maintain separate memory for that list. |
Shifting gears to #504. |
Need to refactor to use #504 function to add page memory. |
Code refactored to use #504 and added code to expand heaps when new 64K pages are assigned in contiguous heaps (allowing for larger than 64K allocations). Currently working on unit tests and should be committed this afternoon. |
I was going to do a pull request Friday afternoon, but once I started running all the tests, slow_test was having issues, so I need to get that fixed. Issue is, not a single line of code I touched should be executed for slow_test. |
When build script checks submodules also check recursive submods
# This is the 1st commit message: various improvements # This is the commit message #2: new hash # This is the commit message #3: fix for script path # This is the commit message #4: fixes # This is the commit message #5: fixes # This is the commit message #6: fixes # This is the commit message #7: fixes # This is the commit message #8: fixes # This is the commit message #9: fixes # This is the commit message #10: fixes # This is the commit message #11: fixes # This is the commit message #12: fixes # This is the commit message #13: fixes # This is the commit message #14: fixes # This is the commit message #15: fixes # This is the commit message #16: fixes # This is the commit message #17: fixes # This is the commit message #18: fixes # This is the commit message #19: fixes # This is the commit message #20: fixes # This is the commit message #21: fixes # This is the commit message #22: fixes # This is the commit message #23: fixes # This is the commit message #24: fixes # This is the commit message #25: fixes # This is the commit message #26: testing # This is the commit message #27: testing # This is the commit message #28: testing # This is the commit message #29: testing # This is the commit message #30: testing # This is the commit message #31: testing # This is the commit message #32: testing # This is the commit message #33: testing # This is the commit message #34: testing # This is the commit message #35: testing # This is the commit message #36: testing # This is the commit message #37: testing # This is the commit message #38: testing # This is the commit message #39: testing # This is the commit message #40: testing # This is the commit message #41: testing # This is the commit message #42: testing # This is the commit message #43: testing # This is the commit message #44: fixes # This is the commit message #45: fixes # This is the commit message #46: fixes # This is the commit message #47: fixes # This is the commit message #48: fixes # This is the commit message #49: fixes # This is the commit message #50: fixes # This is the commit message #51: fixes # This is the commit message #52: fixes # This is the commit message #53: fixes # This is the commit message #54: fixes # This is the commit message #55: fixes # This is the commit message #56: fixes # This is the commit message #57: fixes # This is the commit message #58: fixes # This is the commit message #59: fixes # This is the commit message #60: fixes # This is the commit message #61: fixes # This is the commit message #62: fixes # This is the commit message #63: fixes # This is the commit message #64: fixes # This is the commit message #65: fixes # This is the commit message #66: fixes # This is the commit message #67: fixes # This is the commit message #68: fixes # This is the commit message #69: fixes # This is the commit message #70: fixes # This is the commit message #71: fixes # This is the commit message #72: fixes # This is the commit message #73: fixes # This is the commit message #74: fixes # This is the commit message #75: fixes # This is the commit message #76: fixes # This is the commit message #77: fixes # This is the commit message #78: fixes # This is the commit message #79: more testing # This is the commit message #80: testing # This is the commit message #81: fixes # This is the commit message #82: fixes # This is the commit message #83: fixes # This is the commit message #84: fixes # This is the commit message #85: fixes # This is the commit message #86: fixes # This is the commit message #87: fixes # This is the commit message #88: fixes # This is the commit message #89: fixes # This is the commit message #90: fixes # This is the commit message #91: fixes # This is the commit message #92: fixes # This is the commit message #93: propagate-environment for buildkite-agent # This is the commit message #94: propagate-environment for buildkite-agent # This is the commit message #95: propagate-environment for buildkite-agent # This is the commit message #96: propagate-environment for buildkite-agent # This is the commit message #97: fixes # This is the commit message #98: fixes # This is the commit message #99: fixes # This is the commit message #100: fixes # This is the commit message #101: fixes # This is the commit message #102: fixes # This is the commit message #103: fixes # This is the commit message #104: fixes # This is the commit message #105: fixes # This is the commit message #106: fixes # This is the commit message #107: fixes # This is the commit message #108: fixes # This is the commit message #109: fixes # This is the commit message #110: fixes # This is the commit message #111: fixes # This is the commit message #112: fixes # This is the commit message #113: fixes # This is the commit message #114: fixes # This is the commit message #115: fixes # This is the commit message #116: fixes # This is the commit message #117: fixes # This is the commit message #118: fixes # This is the commit message #119: fixes # This is the commit message #120: fixes # This is the commit message #121: fixes # This is the commit message #122: fixes # This is the commit message #123: fixes # This is the commit message #124: fixes # This is the commit message #125: fixes # This is the commit message #126: fixes # This is the commit message #127: fixes # This is the commit message #128: fixes # This is the commit message #129: fixes # This is the commit message #130: fixes # This is the commit message #131: fixes # This is the commit message #132: fixes # This is the commit message #133: fixes # This is the commit message #134: fixes # This is the commit message #135: fixes # This is the commit message #136: fixes # This is the commit message #137: fixes # This is the commit message #138: fixes # This is the commit message #139: fixes # This is the commit message #140: fixes # This is the commit message #141: fixes # This is the commit message #142: fixes # This is the commit message #143: fixes # This is the commit message #144: fixes # This is the commit message #145: fixes # This is the commit message #146: fixes # This is the commit message #147: fixes # This is the commit message #148: fixes # This is the commit message #149: fixes # This is the commit message #150: fixes # This is the commit message #151: fixes # This is the commit message #152: fixes # This is the commit message #153: testing # This is the commit message #154: fixes # This is the commit message #155: fixes # This is the commit message #156: fixes # This is the commit message #157: fixes # This is the commit message #158: fixes # This is the commit message #159: fixes # This is the commit message #160: fixes # This is the commit message #161: fixes # This is the commit message #162: fixes # This is the commit message #163: fixes # This is the commit message #164: fixes # This is the commit message #165: fixes # This is the commit message #166: fixes # This is the commit message #167: fixes # This is the commit message #168: fixes # This is the commit message #169: fixes # This is the commit message #170: fixes # This is the commit message #171: fixes # This is the commit message #172: fixes # This is the commit message #173: fixes # This is the commit message #174: fixes # This is the commit message #175: fixes # This is the commit message #176: fixes # This is the commit message #177: fixes # This is the commit message #178: fixes # This is the commit message #179: fixes # This is the commit message #180: fixes # This is the commit message #181: fixes # This is the commit message #182: fixes # This is the commit message #183: fixes # This is the commit message #184: fixes # This is the commit message #185: fixes # This is the commit message #186: fixes
The basic idea is that we should make it easy for developers to do some dynamic memory allocation for certain applications.
Currently a process is limited to 64KB of memory including stack and static data. This memory limit creates a defacto limit on the size of a generated transaction which would likely be less than 32KB.
The way I envision malloc being implemented is via a global static char[] buffer that uses pre-processor macro to define the reserved memory.
We need to ensure that compiled programs do not attempt to reserve more than the 64KB, including the reserved stack space.
64KB should be enough for anybody... :)
The text was updated successfully, but these errors were encountered: