feat(libs,metagen): draft python

wip: python mdk feat: represent union, either with Union feat: solve cycles, order python types by priority fix: self-reference fix: premature checks docs: add a comparison b/n metatype and other similar solutions/products. (#697)   - Adds a comparison table between metatype and other similar services. - Add artifact upload protocol to `Architecture` section in docs.   [MET-443](https://linear.app/metatypedev/issue/MET-443/include-comparisons-with-other-products-similar-to-metatype)  _No Migration Needed_  refactor(gate): wasi 0.2 pyrt (#687) - Rewrites the PythonRuntime host using a `componentize-py` based component. - Leaf through this [memo](https://hackmd.io/@SC-qT-WXTROceKYdNA-Lpg/ryyAXiQlC/edit) for a mental model. Todo: - [x] `PythonRuntime.import_` support #699 - [x] ~~Add `pyrt.wasm` to release job~~ obviated by `build.rs` Items for other PRs: - Implemen `hostcall` MET-404. _No end-user changes required_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [ ] End-user documentation is updated to reflect the change --------- Co-authored-by: afmika <afmichael73@gmail.com> cleanups fix: pre-commit fix: forward-refs fix: top level simple types test(metagen): python mdk cleanups and small fixes feat(metagen): codegen decorator and some cleanups feat(metagen): always merge defs if refered file is the same fix(metagen): prioritize relto typegraph path only if base.path is empty feat: Artifact removal (#668) - Add GC: remove artifacts when unreferenced by any deployed typegraph - Improve resource management: use `AsyncDisposable` and `AsyncDisposableStack` - Improve testability (for parallel testing): always read the tmpDir config from the `Typegate` object [MET-433](https://linear.app/metatypedev/issue/MET-433/file-removal) _N/A_ - [x] The change come with new or modified tests - [x] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change  - **New Features** - Enhanced search functionality with the addition of a new search bar. - Introduced new test configurations to improve script execution. - Updated artifact storage documentation to clarify management processes. - Added new extensions to support improved code commenting. - **Bug Fixes** - Removed outdated Deno import mapping settings to streamline development environment setup. - **Documentation** - Expanded documentation on artifact tracking and management, including reference counting and garbage collection mechanisms. - **Refactor** - Implemented interface changes in `QueryEngine` for better async disposal management. - Code restructuring in artifact management for enhanced performance and maintainability. - **Chores** - Adjusted settings and configurations in the development environment to align with current best practices. - **Tests** - Introduced new test cases for artifact upload and management functionalities.  --------- Signed-off-by: Natoandro <anatoandro@hotmail.com> Co-authored-by: destifo <estifanosbireda@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> chore(release): prepare 0.4.0 (#710) Bumps version to release 0.4.0.  - **New Features** - Updated the software across various components to version 0.4.0, enhancing functionality and potentially introducing new features or fixes. - **Documentation** - Updated version documentation in multiple configuration files to reflect new version 0.4.0. - **Bug Fixes** - Adjusted version constants and dependencies to ensure compatibility and stability with the new software version 0.4.0.  --------- Co-authored-by: Natoandro <anatoandro@hotmail.com> chore(docs): final polish to comparison table. (#709) some changes to comparison table(docs) _No Migrations Needed_   - **Documentation** - Introduced a new section on Artifact Tracking Protocol in the architecture documentation, explaining artifact classification and tracking modes in Metatype. - Updated comparisons documentation with additional platforms, criteria for choosing Metatype, and detailed feature comparison tables. - Renamed project directory for clarity and consistency in project setup documentation. - **Bug Fixes** - Removed outdated `TODO` comment in installation documentation.  chore: bump to version 0.4.1-0 (#713) - Bumps version to 0.4.1-0. - Fixes broken release CI. - #719 - Adds 20 minutes to test-full timeout.   - **New Features** - Updated platform support for better compatibility with "x86_64-linux". - **Bug Fixes** - Minor version updates across multiple configurations to enhance stability. - **Chores** - Updated version numbers from "0.4.0" to "0.4.1-0" across various files and configurations. - **Refactor** - Adjusted build and test scripts for improved efficiency and compatibility. - **Documentation** - Enhanced internal documentation to reflect version and platform changes.  --------- Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com> feat: polish documentation and project (#696)   - update the headline, the overviews and many other documentation areas - upgrades the dependencies.   - [ ] The change come with new or modified tests - [ ] Hard-to-understand functions have explanatory comments - [x] End-user documentation is updated to reflect the change  - **Bug Fixes** - Updated Docker image version for the `typegate` service to ensure stability and compatibility. - **Documentation** - Revised `TAGLINE` for better clarity on supported languages: WASM, Typescript, and Python. - Updated version declarations for improved consistency and functionality across multiple files.  --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> feat(sdk): expose metagen to `typegraph/sdk` (#718) Expose metagen features to `typegraph/sdk` Depends on #707 and #696 None  - **New Features** - Introduced new `Metagen` class for code generation tasks in both TypeScript and Python SDKs. - Added functionality for defining policies and structures for deployment examples using Node.js. - **Improvements** - Enhanced `Metagen` class with methods for simulating and executing code generation tasks. - Simplified file reading and writing functions for better performance and maintainability. - **Bug Fixes** - Refined `compress_and_encode` function to streamline file handling processes. - **Tests** - Added comprehensive tests for `Metagen` functionality in both TypeScript and Python SDKs.  --------- Signed-off-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: Teo Stocco <teo@zifeo.com> Co-authored-by: Teo Stocco <zifeo@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Estifanos Bireda <77430541+destifo@users.noreply.github.com> Co-authored-by: Yohe-Am <56622350+Yohe-Am@users.noreply.github.com>
metatypedev · May 22, 2024 · 7d9106e · 7d9106e
1 parent b6788a6
commit 7d9106e
Show file tree

Hide file tree

Showing 37 changed files with 2,531 additions and 181 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/examples/deploy/deploy.mjs b/examples/deploy/deploy.mjs
@@ -62,7 +62,7 @@ const tg = await typegraph({
       },
     ),
     // Wasm
-    testWasmAdd: wasm.fromWasm(
+    testWasmAdd: wasm.fromExport(
       t.struct({ a: t.float(), b: t.float() }),
       t.integer(),
       { wasm: "wasm/rust.wasm", func: "add" },

diff --git a/examples/deploy/deploy.py b/examples/deploy/deploy.py
@@ -59,7 +59,7 @@ def deploy_example_python(g: Graph):
             name="sayHello",
         ),
         # Wasm
-        testWasmAdd=wasm.from_wasm(
+        testWasmAdd=wasm.from_export(
             t.struct({"a": t.float(), "b": t.float()}),
             t.integer(),
             func="add",

diff --git a/libs/common/Cargo.toml b/libs/common/Cargo.toml
@@ -8,7 +8,6 @@ anyhow.workspace = true
 base64 = "0.21.5"
 flate2 = "1.0.28"
 indexmap.workspace = true
-schemars = { version = "0.8.16", features = ["derive", "preserve_order"], optional = true }
 serde.workspace = true
 serde_json = { workspace = true, features = ["preserve_order"] }
 serde_with = "3.4.0"
@@ -20,6 +19,3 @@ itertools = "0.11.0"
 colored = "2.0.4"
 indoc.workspace = true
 thiserror.workspace = true
-
-[features]
-codegen = ["dep:schemars"]
diff --git a/libs/metagen/Cargo.toml b/libs/metagen/Cargo.toml
@@ -8,7 +8,7 @@ common.workspace = true
 log.workspace = true
 serde.workspace = true
 serde_json.workspace = true
-tokio = { workspace = true, features =["rt-multi-thread"]}
+tokio = { workspace = true, features =["rt-multi-thread"], optional = true }
 indexmap.workspace = true
 reqwest = { workspace = true, features = ["json"] }
 garde = { version = "0.18", features = ["derive"] }
@@ -18,7 +18,11 @@ once_cell.workspace = true
 pretty_assertions = "1.4.0"
 color-eyre.workspace = true
 # indoc.workspace = true
+tera = { version = "1", default-features = false }
 
 [dev-dependencies]
-tokio = { workspace = true, features =["full"]}
+tokio = { workspace = true, features =["full"] }
 tempfile.workspace = true
+
+[features]
+multithreaded = ["dep:tokio"]
diff --git a/libs/metagen/src/lib.rs b/libs/metagen/src/lib.rs
@@ -24,6 +24,7 @@ mod interlude {
 
 mod config;
 mod mdk;
+mod mdk_python;
 mod mdk_rust;
 #[cfg(test)]
 mod tests;
@@ -62,6 +63,11 @@ pub trait InputResolver {
     ) -> impl std::future::Future<Output = anyhow::Result<GeneratorInputResolved>> + Send;
 }
 
+/// This type plays the "dispatcher" role to the command object
+pub trait InputResolverSync {
+    fn resolve(&self, order: GeneratorInputOrder) -> anyhow::Result<GeneratorInputResolved>;
+}
+
 #[derive(Debug)]
 pub struct GeneratedFile {
     // pub path: PathBuf,
@@ -85,30 +91,61 @@ trait Plugin: Send + Sync {
     ) -> anyhow::Result<GeneratorOutput>;
 }
 
+type PluginOutputResult = Result<Box<dyn Plugin>, anyhow::Error>;
+
+#[derive(Clone)]
+struct GeneratorRunner {
+    pub op: fn(&Path, serde_json::Value) -> PluginOutputResult,
+}
+
+impl GeneratorRunner {
+    pub fn exec(&self, workspace_path: &Path, value: serde_json::Value) -> PluginOutputResult {
+        (self.op)(workspace_path, value)
+    }
+}
+
+thread_local! {
+    static GENERATORS: HashMap<String, GeneratorRunner> = HashMap::from([
+        // builtin generators
+        (
+            "mdk_rust".to_string(),
+            GeneratorRunner {
+                op: |workspace_path: &Path, val| {
+                    let config = mdk_rust::MdkRustGenConfig::from_json(val, workspace_path)?;
+                    let generator = mdk_rust::Generator::new(config)?;
+                    Ok(Box::new(generator))
+                },
+            },
+        ),
+        (
+            "mdk_python".to_string(),
+            GeneratorRunner {
+                op: |workspace_path: &Path, val| {
+                    let config = mdk_python::MdkPythonGenConfig::from_json(val, workspace_path)?;
+                    let generator = mdk_python::Generator::new(config)?;
+                    Ok(Box::new(generator))
+                },
+            },
+        ),
+    ]);
+}
+
+impl GeneratorRunner {
+    pub fn get(name: &str) -> Option<GeneratorRunner> {
+        GENERATORS.with(|m| m.get(name).cloned())
+    }
+}
+
 /// This function makes use of a JoinSet to process
 /// items in parallel. This makes using actix workers in InputResolver
 /// is a no no.
+#[cfg(feature = "multithreaded")]
 pub async fn generate_target(
     config: &config::Config,
     target_name: &str,
     workspace_path: PathBuf,
     resolver: impl InputResolver + Send + Sync + Clone + 'static,
 ) -> anyhow::Result<GeneratorOutput> {
-    let generators = [
-        // builtin generators
-        (
-            "mdk_rust".to_string(),
-            // initialize the impl
-            &|workspace_path: &Path, val| {
-                let config = mdk_rust::MdkRustGenConfig::from_json(val, workspace_path)?;
-                let generator = mdk_rust::Generator::new(config)?;
-                Ok::<_, anyhow::Error>(Box::new(generator) as Box<dyn Plugin>)
-            },
-        ),
-    ]
-    .into_iter()
-    .collect::<HashMap<String, _>>();
-
     let target_conf = config
         .targets
         .get(target_name)
@@ -118,11 +155,10 @@ pub async fn generate_target(
     for (gen_name, config) in &target_conf.0 {
         let config = config.to_owned();
 
-        let get_gen_fn = generators
-            .get(&gen_name[..])
-            .with_context(|| format!("generator {gen_name:?} not found in config"))?;
+        let get_gen_op = GeneratorRunner::get(gen_name)
+            .with_context(|| format!("generator \"{gen_name}\" not found in config"))?;
 
-        let gen_impl = get_gen_fn(&workspace_path, config)?;
+        let gen_impl = get_gen_op.exec(&workspace_path, config)?;
         let bill = gen_impl.bill_of_inputs();
 
         let mut resolve_set = tokio::task::JoinSet::new();
@@ -154,6 +190,61 @@ pub async fn generate_target(
             out.insert(path, (gen_name.clone(), buf));
         }
     }
+    let out: HashMap<PathBuf, GeneratedFile> = out
+        .into_iter()
+        .map(|(path, (_, buf))| (path, buf))
+        .collect();
+    Ok(GeneratorOutput(out))
+}
+
+pub fn generate_target_sync(
+    config: &config::Config,
+    target_name: &str,
+    workspace_path: PathBuf,
+    resolver: impl InputResolverSync + Send + Sync + Clone + 'static,
+) -> anyhow::Result<GeneratorOutput> {
+    let target_conf = config
+        .targets
+        .get(target_name)
+        .with_context(|| format!("target \"{target_name}\" not found in config"))?;
+
+    let mut generate_set = vec![];
+    for (gen_name, config) in &target_conf.0 {
+        let config = config.to_owned();
+
+        let get_gen_op = GeneratorRunner::get(gen_name)
+            .with_context(|| format!("generator \"{gen_name}\" not found in config"))?;
+
+        let gen_impl = get_gen_op.exec(&workspace_path, config)?;
+        let bill = gen_impl.bill_of_inputs();
+
+        let resolve_set = bill.into_iter().map(|(name, order)| {
+            let resolver = resolver.clone();
+            Ok::<_, anyhow::Error>((name, resolver.resolve(order)))
+        });
+
+        let gen_name: Arc<str> = gen_name[..].into();
+        generate_set.push(move || {
+            let mut inputs = HashMap::new();
+            for res in resolve_set {
+                let (name, input) = res?;
+                inputs.insert(name, input?);
+            }
+            let out = gen_impl.generate(inputs)?;
+            Ok::<_, anyhow::Error>((gen_name, out))
+        });
+    }
+
+    let mut out = HashMap::new();
+    for res in generate_set {
+        let (gen_name, files) = res()?;
+        for (path, buf) in files.0 {
+            if let Some((src, _)) = out.get(&path) {
+                anyhow::bail!("generators \"{src}\" and \"{gen_name}\" clashed at \"{path:?}\"");
+            }
+            out.insert(path, (gen_name.clone(), buf));
+        }
+    }
     let out = out
         .into_iter()
         .map(|(path, (_, buf))| (path, buf))