refactor(gate): wasi 0.2 pyrt #687
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Yohe-Am
commented
Apr 21, 2024
Yohe-Am
force-pushed
the
met-447-gate-wasmtime-support
branch
from
929c5d3 to
3c43717
Compare
Yohe-Am
force-pushed
the
refactor/MET-404/wasmtime-pyrt
branch
from
6b10f36 to
876db95
Compare
3 tasks
Yohe-Am
force-pushed
the
refactor/MET-404/wasmtime-pyrt
branch
from
876db95 to
97e34c4
Compare
zifeo
reviewed
Apr 22, 2024
Yohe-Am
force-pushed
the
refactor/MET-404/wasmtime-pyrt
branch
3 times, most recently
from
29b35a1 to
30ff471
Compare
This was referenced Apr 26, 2024
fix: cherry-pick feat(wasm): wit component support test(wasm): error propagation refactor(wasm): all remaining wasmedge -> wasm fix(wasm): bad conversion feat(wasm): nested object output test(wasm): tuple deserialize test(wasm): update binding test value feat(wasm): nested wit input support feat(wasm): handle enum input, fix object bug feat(wasm): reject on extra fields fix(wasm): typos and minor cleanups fix(tests.yml): disable cache deno dir for now fix: version lock fix: lockfile refactor: rename all wasmedge ref. to wasm
Yohe-Am
force-pushed
the
refactor/MET-404/wasmtime-pyrt
branch
from
30ff471 to
93e1a7d
Compare
Yohe-Am
added a commit
that referenced
this pull request
Apr 30, 2024
Implements python runtime support for `import_function` materializers. This is an stacked PR on #673 which itself is appended on #687 for tackling MET-404. #### Migration notes This PR introduced a breaking change in the importer logic for python modules brought in through `pyrt.import_` and it's deps. After this PR, any path imports must be prefixed with `.`, `..`, `...` and so to point to relative path locations. This behavior is at odds with standard python which looks for relative modules and packages even if there are no dot prefixes. - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments <!-- 5. Readiness checklist - [ ] End-user documentation is updated to reflect the change -->
- Finalizes the host side of the `wit_wire` implementation started in #687 . - Fixes a few issues with the `mdk_rust` generator. #### Motivation and context The code generator was already in place but we the typegate had no support for the `wit_wire` interface used by the rust mdk. This PR adds that. The two week delay is mainly due to the base work required in #669 and related PRs. #### Migration notes _No breaking changes on user._ ### Checklist - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change
zifeo
previously approved these changes
May 1, 2024
|
Important Auto Review SkippedMore than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review. 55 files out of 121 files are above the max files limit of 50. Please upgrade to Pro plan to get higher limits. You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #687 +/- ##
==========================================
- Coverage 75.69% 73.95% -1.74%
==========================================
Files 121 121
Lines 13583 13555 -28
Branches 1415 1383 -32
==========================================
- Hits 10281 10024 -257
- Misses 3279 3507 +228
- Partials 23 24 +1 ☔ View full report in Codecov by Sentry. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
destifo
approved these changes
May 7, 2024
Yohe-Am
added a commit
that referenced
this pull request
May 22, 2024
wip: python mdk feat: represent union, either with Union feat: solve cycles, order python types by priority fix: self-reference fix: premature checks docs: add a comparison b/n metatype and other similar solutions/products. (#697) <!-- Pull requests are squash merged using: - their title as the commit message - their description as the commit body Having a good title and description is important for the users to get readable changelog. --> <!-- 1. Explain below WHAT the change is --> - Adds a comparison table between metatype and other similar services. - Add artifact upload protocol to `Architecture` section in docs. <!-- 2. Explain below WHY the change cannot be made simpler --> <!-- 3. Explain below WHY the was made or link an issue number --> [MET-443](https://linear.app/metatypedev/issue/MET-443/include-comparisons-with-other-products-similar-to-metatype) <!-- 4. Explain HOW users should update their code or remove that section --> _No Migration Needed_ <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> refactor(gate): wasi 0.2 pyrt (#687) - Rewrites the PythonRuntime host using a `componentize-py` based component. - Leaf through this [memo](https://hackmd.io/@SC-qT-WXTROceKYdNA-Lpg/ryyAXiQlC/edit) for a mental model. Todo: - [x] `PythonRuntime.import_` support #699 - [x] ~~Add `pyrt.wasm` to release job~~ obviated by `build.rs` Items for other PRs: - Implemen `hostcall` MET-404. _No end-user changes required_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --------- Co-authored-by: afmika <afmichael73@gmail.com> cleanups fix: pre-commit fix: forward-refs fix: top level simple types test(metagen): python mdk cleanups and small fixes feat(metagen): codegen decorator and some cleanups feat(metagen): always merge defs if refered file is the same fix(metagen): prioritize relto typegraph path only if base.path is empty feat: Artifact removal (#668) - Add GC: remove artifacts when unreferenced by any deployed typegraph - Improve resource management: use `AsyncDisposable` and `AsyncDisposableStack` - Improve testability (for parallel testing): always read the tmpDir config from the `Typegate` object [MET-433](https://linear.app/metatypedev/issue/MET-433/file-removal) _N/A_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Enhanced search functionality with the addition of a new search bar. - Introduced new test configurations to improve script execution. - Updated artifact storage documentation to clarify management processes. - Added new extensions to support improved code commenting. - **Bug Fixes** - Removed outdated Deno import mapping settings to streamline development environment setup. - **Documentation** - Expanded documentation on artifact tracking and management, including reference counting and garbage collection mechanisms. - **Refactor** - Implemented interface changes in `QueryEngine` for better async disposal management. - Code restructuring in artifact management for enhanced performance and maintainability. - **Chores** - Adjusted settings and configurations in the development environment to align with current best practices. - **Tests** - Introduced new test cases for artifact upload and management functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Natoandro <anatoandro@hotmail.com> Co-authored-by: destifo <estifanosbireda@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> chore(release): prepare 0.4.0 (#710) Bumps version to release 0.4.0. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Updated the software across various components to version 0.4.0, enhancing functionality and potentially introducing new features or fixes. - **Documentation** - Updated version documentation in multiple configuration files to reflect new version 0.4.0. - **Bug Fixes** - Adjusted version constants and dependencies to ensure compatibility and stability with the new software version 0.4.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Natoandro <anatoandro@hotmail.com> chore(docs): final polish to comparison table. (#709) some changes to comparison table(docs) _No Migrations Needed_ <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **Documentation** - Introduced a new section on Artifact Tracking Protocol in the architecture documentation, explaining artifact classification and tracking modes in Metatype. - Updated comparisons documentation with additional platforms, criteria for choosing Metatype, and detailed feature comparison tables. - Renamed project directory for clarity and consistency in project setup documentation. - **Bug Fixes** - Removed outdated `TODO` comment in installation documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> chore: bump to version 0.4.1-0 (#713) - Bumps version to 0.4.1-0. - Fixes broken release CI. - #719 - Adds 20 minutes to test-full timeout. <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Updated platform support for better compatibility with "x86_64-linux". - **Bug Fixes** - Minor version updates across multiple configurations to enhance stability. - **Chores** - Updated version numbers from "0.4.0" to "0.4.1-0" across various files and configurations. - **Refactor** - Adjusted build and test scripts for improved efficiency and compatibility. - **Documentation** - Enhanced internal documentation to reflect version and platform changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com> feat: polish documentation and project (#696) <!-- Pull requests are squash merged using: - their title as the commit message - their description as the commit body Having a good title and description is important for the users to get readable changelog. --> <!-- 1. Explain below WHAT the change is --> - update the headline, the overviews and many other documentation areas - upgrades the dependencies. <!-- 2. Explain below WHY the change cannot be made simpler --> <!-- 4. Explain HOW users should update their code or remove that section --> - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **Bug Fixes** - Updated Docker image version for the `typegate` service to ensure stability and compatibility. - **Documentation** - Revised `TAGLINE` for better clarity on supported languages: WASM, Typescript, and Python. - Updated version declarations for improved consistency and functionality across multiple files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> feat(sdk): expose metagen to `typegraph/sdk` (#718) Expose metagen features to `typegraph/sdk` Depends on #707 and #696 None <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Introduced new `Metagen` class for code generation tasks in both TypeScript and Python SDKs. - Added functionality for defining policies and structures for deployment examples using Node.js. - **Improvements** - Enhanced `Metagen` class with methods for simulating and executing code generation tasks. - Simplified file reading and writing functions for better performance and maintainability. - **Bug Fixes** - Refined `compress_and_encode` function to streamline file handling processes. - **Tests** - Added comprehensive tests for `Metagen` functionality in both TypeScript and Python SDKs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: Teo Stocco <teo@zifeo.com> Co-authored-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Estifanos Bireda <77430541+destifo@users.noreply.github.com> Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com>
Yohe-Am
added a commit
that referenced
this pull request
May 22, 2024
wip: python mdk feat: represent union, either with Union feat: solve cycles, order python types by priority fix: self-reference fix: premature checks docs: add a comparison b/n metatype and other similar solutions/products. (#697) <!-- Pull requests are squash merged using: - their title as the commit message - their description as the commit body Having a good title and description is important for the users to get readable changelog. --> <!-- 1. Explain below WHAT the change is --> - Adds a comparison table between metatype and other similar services. - Add artifact upload protocol to `Architecture` section in docs. <!-- 2. Explain below WHY the change cannot be made simpler --> <!-- 3. Explain below WHY the was made or link an issue number --> [MET-443](https://linear.app/metatypedev/issue/MET-443/include-comparisons-with-other-products-similar-to-metatype) <!-- 4. Explain HOW users should update their code or remove that section --> _No Migration Needed_ <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> refactor(gate): wasi 0.2 pyrt (#687) - Rewrites the PythonRuntime host using a `componentize-py` based component. - Leaf through this [memo](https://hackmd.io/@SC-qT-WXTROceKYdNA-Lpg/ryyAXiQlC/edit) for a mental model. Todo: - [x] `PythonRuntime.import_` support #699 - [x] ~~Add `pyrt.wasm` to release job~~ obviated by `build.rs` Items for other PRs: - Implemen `hostcall` MET-404. _No end-user changes required_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --------- Co-authored-by: afmika <afmichael73@gmail.com> cleanups fix: pre-commit fix: forward-refs fix: top level simple types test(metagen): python mdk cleanups and small fixes feat(metagen): codegen decorator and some cleanups feat(metagen): always merge defs if refered file is the same fix(metagen): prioritize relto typegraph path only if base.path is empty feat: Artifact removal (#668) - Add GC: remove artifacts when unreferenced by any deployed typegraph - Improve resource management: use `AsyncDisposable` and `AsyncDisposableStack` - Improve testability (for parallel testing): always read the tmpDir config from the `Typegate` object [MET-433](https://linear.app/metatypedev/issue/MET-433/file-removal) _N/A_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Enhanced search functionality with the addition of a new search bar. - Introduced new test configurations to improve script execution. - Updated artifact storage documentation to clarify management processes. - Added new extensions to support improved code commenting. - **Bug Fixes** - Removed outdated Deno import mapping settings to streamline development environment setup. - **Documentation** - Expanded documentation on artifact tracking and management, including reference counting and garbage collection mechanisms. - **Refactor** - Implemented interface changes in `QueryEngine` for better async disposal management. - Code restructuring in artifact management for enhanced performance and maintainability. - **Chores** - Adjusted settings and configurations in the development environment to align with current best practices. - **Tests** - Introduced new test cases for artifact upload and management functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Natoandro <anatoandro@hotmail.com> Co-authored-by: destifo <estifanosbireda@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> chore(release): prepare 0.4.0 (#710) Bumps version to release 0.4.0. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Updated the software across various components to version 0.4.0, enhancing functionality and potentially introducing new features or fixes. - **Documentation** - Updated version documentation in multiple configuration files to reflect new version 0.4.0. - **Bug Fixes** - Adjusted version constants and dependencies to ensure compatibility and stability with the new software version 0.4.0. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Natoandro <anatoandro@hotmail.com> chore(docs): final polish to comparison table. (#709) some changes to comparison table(docs) _No Migrations Needed_ <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **Documentation** - Introduced a new section on Artifact Tracking Protocol in the architecture documentation, explaining artifact classification and tracking modes in Metatype. - Updated comparisons documentation with additional platforms, criteria for choosing Metatype, and detailed feature comparison tables. - Renamed project directory for clarity and consistency in project setup documentation. - **Bug Fixes** - Removed outdated `TODO` comment in installation documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> chore: bump to version 0.4.1-0 (#713) - Bumps version to 0.4.1-0. - Fixes broken release CI. - #719 - Adds 20 minutes to test-full timeout. <!-- 5. Readiness checklist - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Updated platform support for better compatibility with "x86_64-linux". - **Bug Fixes** - Minor version updates across multiple configurations to enhance stability. - **Chores** - Updated version numbers from "0.4.0" to "0.4.1-0" across various files and configurations. - **Refactor** - Adjusted build and test scripts for improved efficiency and compatibility. - **Documentation** - Enhanced internal documentation to reflect version and platform changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com> feat: polish documentation and project (#696) <!-- Pull requests are squash merged using: - their title as the commit message - their description as the commit body Having a good title and description is important for the users to get readable changelog. --> <!-- 1. Explain below WHAT the change is --> - update the headline, the overviews and many other documentation areas - upgrades the dependencies. <!-- 2. Explain below WHY the change cannot be made simpler --> <!-- 4. Explain HOW users should update their code or remove that section --> - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **Bug Fixes** - Updated Docker image version for the `typegate` service to ensure stability and compatibility. - **Documentation** - Revised `TAGLINE` for better clarity on supported languages: WASM, Typescript, and Python. - Updated version declarations for improved consistency and functionality across multiple files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> feat(sdk): expose metagen to `typegraph/sdk` (#718) Expose metagen features to `typegraph/sdk` Depends on #707 and #696 None <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Introduced new `Metagen` class for code generation tasks in both TypeScript and Python SDKs. - Added functionality for defining policies and structures for deployment examples using Node.js. - **Improvements** - Enhanced `Metagen` class with methods for simulating and executing code generation tasks. - Simplified file reading and writing functions for better performance and maintainability. - **Bug Fixes** - Refined `compress_and_encode` function to streamline file handling processes. - **Tests** - Added comprehensive tests for `Metagen` functionality in both TypeScript and Python SDKs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: Teo Stocco <teo@zifeo.com> Co-authored-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Estifanos Bireda <77430541+destifo@users.noreply.github.com> Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com>
Yohe-Am
added a commit
that referenced
this pull request
May 23, 2024
Introduces a mechanism for wasm materializers to access hostgate functions. This implements a pretty basic JSON wire interface, a singular `hostcall` function that's exposed to materializers. The only implemented function on this interface are `gql` queries. This is a stacked PR on top of #687. MET-473. - [x] The change come with new or modified tests <!-- 5. Readiness checklist - [ ] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added an import statement for `std_url` and a new task for installing WASI adapter related files. - Introduced new functionalities in the application's runtime to support additional parameters and error handling. - **Enhancements** - Improved the application's handling of GraphQL queries with new error types and display methods. - Enhanced the WASM runtime build process to target a more appropriate architecture. - **Bug Fixes** - Fixed issues in Python and WASM runtime tests to ensure reliability and performance. - **Documentation** - Updated internal documentation to reflect new command interfaces and environmental interactions in the application's CLI tools. - **Refactor** - Refactored various internal APIs to improve code maintainability and efficiency. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: afmika <afmichael73@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
componentize-pybased component.Todo:
PythonRuntime.import_support feat(pyrt)!:import_functionmaterializer support #699Addobviated bypyrt.wasmto release jobbuild.rsItems for other PRs:
hostcallMotivation and context
MET-404.
Migration notes
No end-user changes required
Checklist