feat(component): rust component model support #128

chrisdickinson · 2023-10-04T00:18:24Z

upgrade to wasmtime and wasmtime_wasi to ">= 17.0.0, < 18.0.0"
- 17 is the wasi preview 2 supporting release
the component model necessitated splitting the adapter setup into two parts: first creating the adapter so it can be associated with the state, then telling the linker how to get from the incoming state to the adapter. WASI also has this split.
change the c bindings in observe-api:
1. "_" is an illegal character! switch to "-". (This is a breaking change!)
2. change the module name from "dylibso_observe" to "dylibso:observe/api"
The component model representation of our format enum doesn't match our core module format – Statsd is 0 in the component model version.

Issue: #142

observe-api/rust/src/lib.rs

corpus/00-component-instr-reactor/Cargo.toml

rust/examples/otel-stdout-components.rs

rust/src/context.rs

corpus/00-component-instr-command/src/main.rs

rust/src/context.rs

G4Vi · 2023-10-23T21:45:17Z

observe-sdk/rust/src/wasm_instr.rs

Line 9 in 5a8d5ac

    
           pub const WASM_INSTR_VERSION_MINOR: u32 = 0; // TODO: bump this to match compiler when ready

should be bumped to 4 to match the change in wasm-instr as changing the naming is a breaking change. This change probably needs to be done on the other language sdks.

The pointers in the observe api were intentionally passed as uint64's in order to have the same api with wasm with 64 bit linear memory. @chrisdickinson what are your thoughts on wasm64 support?

chrisdickinson · 2023-10-24T01:14:27Z

@chrisdickinson what are your thoughts on wasm64 support?

I ended up aligning our API with the component model's canonical ABI, which is defined in terms of 32-bit pointers and extents. It looks like they're tracking the memory64 proposal here and here; but both issues haven't had activity in about a year.

This makes the observe-api-crate-punning approach a little more appealing compared to the component::bindgen approach, to be honest. At least with the crate we could hide the difference so that users wouldn't have to recompile their applications when memory64 support lands in the component model. I'd want to get a read from someone closer to the component model implementation on how likely ABI changes to accommodate memory64 are, though.

G4Vi · 2023-10-24T18:37:39Z

Thanks for the thorough explanation on the pointer passing change! That sounds reasonable.

- upgrade to wasmtime>=13, wasmtime_wasi>=13 - the component model necessitated splitting the adapter setup into two parts: first creating the adapter so it can be associated with the state, then telling the linker how to get from the incoming state to the adapter. WASI also has this split. - change the c bindings in observe-api: 1. "_" is an illegal character! switch to "-". 2. change the module name from "dylibso_observe" to "dylibso:observe/api"

Align core module instrumentation naming with component model naming. This relies on corresponding (undeployed, private) changes to the observe http api [1]. [1]: dylibso/wasm-instr#69

This also removes the WIT `dylibso:observe` world since we don't *really* need it.

Update vendored instrumented wasm modules to match new naming. Additionally: switch github workflow from `npm i` to the (much faster) `npm ci`.

… dylibso_observe namespace

Please use test files in the `test` directory in the root of the project instead

…_observe, with warning. Error on modules instrumented with observe_api

G4Vi · 2024-01-18T23:52:11Z

@mhmd-azeez Could you please try out 0b43202 with the modules and adapters used on #152 ? I just finished rebasing this on top of #152 and manually reapplying my changes to the go sdk and want to be sure it's still working as intended. The only change when running modules with the dylibso_observe namespace should be a deprecation warning printed to stdout:

log.Println("Module uses deprecated namespace \"dylibso_observe\"!\n" +
					"Please consider reinstrumenting with newer wasm-instr or Observe API!")

mhmd-azeez · 2024-01-23T10:39:39Z

@G4Vi sorry, just saw your comment. It seems to be working correctly:

Old modules:

PS D:\dylibso\observe-sdk\go\bin\otelstdout> go run . "D:\x\observe-sdk\observe-api\test\rust_guest.wasm" | jq
2024/01/23 13:37:47 Module uses deprecated namespace "dylibso_observe"!
Please consider reinstrumenting with newer wasm-instr or Observe API!
2024/01/23 13:37:47 metric: worlds.helloed:1|c
2024/01/23 13:37:47 Hello World 1
2024/01/23 13:37:47 metric: worlds.helloed:1|c
2024/01/23 13:37:47 Hello World 2
2024/01/23 13:37:47 metric: worlds.helloed:1|c
2024/01/23 13:37:47 Hello World 3
{
  "resource_spans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "Value": {
                "StringValue": "golang"
              }
            }
          }
        ]
      },
      "scope_spans": [
        {
          "spans": [
            {
              "trace_id": "IBIj3rV2n1slwYUSDAYlhQ==",
              "span_id": "wY/dyqn58kk=",
              "name": "main",
              "kind": 1,
              "start_time_unix_nano": 1706006267423033100,
              "end_time_unix_nano": 1706006267423033100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-main"
                    }
                  }
                },
                {
                  "key": "user_id",
                  "value": {
                    "Value": {
                      "StringValue": "123"
                    }
                  }
                },
                {
                  "key": "world",
                  "value": {
                    "Value": {
                      "StringValue": "hello"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "IBIj3rV2n1slwYUSDAYlhQ==",
              "span_id": "FaQZ7o/bA2E=",
              "parent_span_id": "wY/dyqn58kk=",
              "name": "run",
              "kind": 1,
              "start_time_unix_nano": 1706006267423033100,
              "end_time_unix_nano": 1706006267423033100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-run"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "IBIj3rV2n1slwYUSDAYlhQ==",
              "span_id": "PtKcp19WCwo=",
              "parent_span_id": "FaQZ7o/bA2E=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006267423033100,
              "end_time_unix_nano": 1706006267423033100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "IBIj3rV2n1slwYUSDAYlhQ==",
              "span_id": "NjHWDxzMqEg=",
              "parent_span_id": "FaQZ7o/bA2E=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006267423033100,
              "end_time_unix_nano": 1706006267423033100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "IBIj3rV2n1slwYUSDAYlhQ==",
              "span_id": "JyCk0E2FLwM=",
              "parent_span_id": "FaQZ7o/bA2E=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006267423033100,
              "end_time_unix_nano": 1706006267423033100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
PS D:\dylibso\observe-sdk\go\bin\otelstdout> cd ..\stdout\
PS D:\dylibso\observe-sdk\go\bin\stdout> go run . "D:\x\observe-sdk\observe-api\test\rust_guest.wasm"
2024/01/23 13:38:19 Module uses deprecated namespace "dylibso_observe"!
Please consider reinstrumenting with newer wasm-instr or Observe API!
2024/01/23 13:38:19 metric: worlds.helloed:1|c
2024/01/23 13:38:19 Hello World 1
2024/01/23 13:38:19 metric: worlds.helloed:1|c
2024/01/23 13:38:19 Hello World 2
2024/01/23 13:38:19 metric: worlds.helloed:1|c
2024/01/23 13:38:19 Hello World 3
2024/01/23 13:38:19  Call to main took 502µs
2024/01/23 13:38:19  Span tags: [user_id:123 world:hello]
2024/01/23 13:38:19    Call to run took 502µs
2024/01/23 13:38:19      Call to log_something took 502µs
2024/01/23 13:38:19      Call to log_something took 0s
2024/01/23 13:38:19      Call to log_something took 0s

New modules:

PS D:\dylibso\observe-sdk\go\bin\stdout> go run . "D:\dylibso\observe-sdk\observe-api\test\rust_guest.wasm"
2024/01/23 13:39:17 metric: worlds.helloed:1|c
2024/01/23 13:39:17 Hello World 1
2024/01/23 13:39:17 metric: worlds.helloed:1|c
2024/01/23 13:39:17 Hello World 2
2024/01/23 13:39:17 metric: worlds.helloed:1|c
2024/01/23 13:39:17 Hello World 3
2024/01/23 13:39:17  Call to main took 0s
2024/01/23 13:39:17  Span tags: [user_id:123 world:hello]
2024/01/23 13:39:17    Call to run took 0s
2024/01/23 13:39:17      Call to log_something took 0s
2024/01/23 13:39:17      Call to log_something took 0s
2024/01/23 13:39:17      Call to log_something took 0s
PS D:\dylibso\observe-sdk\go\bin\stdout> cd ..\otelstdout\
PS D:\dylibso\observe-sdk\go\bin\otelstdout> go run . "D:\dylibso\observe-sdk\observe-api\test\rust_guest.wasm" | jq
2024/01/23 13:39:31 metric: worlds.helloed:1|c
2024/01/23 13:39:31 Hello World 1
2024/01/23 13:39:31 metric: worlds.helloed:1|c
2024/01/23 13:39:31 Hello World 2
2024/01/23 13:39:31 metric: worlds.helloed:1|c
2024/01/23 13:39:31 Hello World 3
{
  "resource_spans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "Value": {
                "StringValue": "golang"
              }
            }
          }
        ]
      },
      "scope_spans": [
        {
          "spans": [
            {
              "trace_id": "KTuK5VqbIhxZtGZYGhrjBQ==",
              "span_id": "hAtU6Is1GG0=",
              "name": "main",
              "kind": 1,
              "start_time_unix_nano": 1706006371741159100,
              "end_time_unix_nano": 1706006371741159100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-main"
                    }
                  }
                },
                {
                  "key": "user_id",
                  "value": {
                    "Value": {
                      "StringValue": "123"
                    }
                  }
                },
                {
                  "key": "world",
                  "value": {
                    "Value": {
                      "StringValue": "hello"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "KTuK5VqbIhxZtGZYGhrjBQ==",
              "span_id": "NDcJDoLD1j8=",
              "parent_span_id": "hAtU6Is1GG0=",
              "name": "run",
              "kind": 1,
              "start_time_unix_nano": 1706006371741159100,
              "end_time_unix_nano": 1706006371741159100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-run"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "KTuK5VqbIhxZtGZYGhrjBQ==",
              "span_id": "PfvaR+U8Q3U=",
              "parent_span_id": "NDcJDoLD1j8=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006371741159100,
              "end_time_unix_nano": 1706006371741159100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "KTuK5VqbIhxZtGZYGhrjBQ==",
              "span_id": "mWvWM3e/Wxw=",
              "parent_span_id": "NDcJDoLD1j8=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006371741159100,
              "end_time_unix_nano": 1706006371741159100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            },
            {
              "trace_id": "KTuK5VqbIhxZtGZYGhrjBQ==",
              "span_id": "v4kawKvAK0k=",
              "parent_span_id": "NDcJDoLD1j8=",
              "name": "log_something",
              "kind": 1,
              "start_time_unix_nano": 1706006371741159100,
              "end_time_unix_nano": 1706006371741159100,
              "attributes": [
                {
                  "key": "function-name",
                  "value": {
                    "Value": {
                      "StringValue": "function-call-log_something"
                    }
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

…codealliance/cargo-component@5cf73a6

chrisdickinson

These updates look great, @G4Vi! Thanks for taking this over the finish line.

zshipko

Nice, read through the changes and it looks good! Definitely seems worth merging now.

chrisdickinson commented Oct 4, 2023

View reviewed changes

chrisdickinson force-pushed the chris/20231003-hack-in-component-support branch from 64ebcbc to deb3e97 Compare October 4, 2023 00:35

chrisdickinson marked this pull request as ready for review October 11, 2023 00:41

wikiwong reviewed Oct 11, 2023

View reviewed changes

corpus/00-component-instr-command/src/main.rs Show resolved Hide resolved

wikiwong reviewed Oct 11, 2023

View reviewed changes

rust/src/context.rs Show resolved Hide resolved

G4Vi marked this pull request as draft November 9, 2023 23:18

G4Vi changed the title ~~feat(component): wip component model support~~ feat(component): rust component model support Nov 9, 2023

G4Vi mentioned this pull request Nov 9, 2023

Rust Component Model Support for Observe SDK / Observe API #142

Closed

7 tasks

G4Vi mentioned this pull request Nov 20, 2023

Component Model Support for Observe SDK #126

Closed

2 tasks

chrisdickinson mentioned this pull request Dec 8, 2023

Proposal: WASI Observability API WebAssembly/wasi-observe#10

Closed

G4Vi force-pushed the chris/20231003-hack-in-component-support branch 2 times, most recently from 70d2b32 to 9def894 Compare January 17, 2024 23:12

chrisdickinson and others added 15 commits January 17, 2024 18:17

feat(component): rename core module imports

ded7465

Align core module instrumentation naming with component model naming. This relies on corresponding (undeployed, private) changes to the observe http api [1]. [1]: dylibso/wasm-instr#69

feat(component): move from AsMut to ObserveSdkView

c6de5a7

This also removes the WIT `dylibso:observe` world since we don't *really* need it.

fix(corpus): rename 00-component-instr-{reactor → command}

7f9bfa5

feat(component): add an example of depending on an instr'd component

95b8d37

doc: add component model integration guide

a76c43f

fix(component): fix broken CI

bddfd60

Update vendored instrumented wasm modules to match new naming. Additionally: switch github workflow from `npm i` to the (much faster) `npm ci`.

fix: bump WASM_INSTR_VERSION_MINOR

924c538

fix: error out when module and observe-sdk versions are not compatible

d5b7a50

chore: improve messsaging with incompat wasm-instr versions

c7c033f

feat: detect and error out on all modules (incl observe api) that use…

e2da6d0

… dylibso_observe namespace

fix: observe sdk compat with observe api

cc0822e

chore: update go-sdk message for incompatible wasm-instr

caec1a6

fix: go-sdk import function names, don't hang on unknown name

43e2d4f

chore: remove go/test/count_vowels.instr.wasm it was outdated

6ab7101

Please use test files in the `test` directory in the root of the project instead

G4Vi added 7 commits January 17, 2024 18:17

feat: js-sdk support old api too

904a323

chore: js-sdk warn on discovery of deprecated namespace dylibso_observe

2b5037a

feat: remove checks on version globals in rust-sdk

8bc38d1

feat: support modules instrumented with deprecrated namespace dylibso…

1830e7d

…_observe, with warning. Error on modules instrumented with observe_api

feat: error out when observe api is used on go and js sdks

62ee10a

docs: add SDK-API-VERSIONING.md to document how to do api changes

86aba1b

fix: error out on when even new observe api is used on go and js sdks

e94c46f

G4Vi force-pushed the chris/20231003-hack-in-component-support branch from 9def894 to e94c46f Compare January 17, 2024 23:18

chore: readd go changes

0b43202

chrisdickinson and others added 6 commits January 24, 2024 11:53

fix: new versions of WIT require semicolons

0652bfe

fix,docs: component building and running, add instructions to README

fa29a81

ci: add testing components

ebfc08d

ci: install just, use action to install rust cli

3abbaa4

chore: bin versions

630bae5

chore: update to wasmtime 17, move off cargo-component-bindings: byte…

d9e3323

…codealliance/cargo-component@5cf73a6

G4Vi marked this pull request as ready for review January 30, 2024 01:01

G4Vi requested review from bhelx, zshipko and nilslice January 30, 2024 01:03

chrisdickinson commented Jan 30, 2024

View reviewed changes

zshipko approved these changes Jan 30, 2024

View reviewed changes

G4Vi merged commit 231702d into main Jan 31, 2024
6 checks passed

G4Vi deleted the chris/20231003-hack-in-component-support branch January 31, 2024 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(component): rust component model support #128

feat(component): rust component model support #128

chrisdickinson commented Oct 4, 2023 •

edited by G4Vi

Loading

G4Vi commented Oct 23, 2023

chrisdickinson commented Oct 24, 2023

G4Vi commented Oct 24, 2023

G4Vi commented Jan 18, 2024

mhmd-azeez commented Jan 23, 2024

chrisdickinson left a comment

zshipko left a comment

feat(component): rust component model support #128

feat(component): rust component model support #128

Conversation

chrisdickinson commented Oct 4, 2023 • edited by G4Vi Loading

G4Vi commented Oct 23, 2023

chrisdickinson commented Oct 24, 2023

G4Vi commented Oct 24, 2023

G4Vi commented Jan 18, 2024

mhmd-azeez commented Jan 23, 2024

chrisdickinson left a comment

Choose a reason for hiding this comment

zshipko left a comment

Choose a reason for hiding this comment

chrisdickinson commented Oct 4, 2023 •

edited by G4Vi

Loading