otel: implement just otel
#2020
Assignees
Comments
Open
This was referenced Jul 9, 2024
deniseli
added a commit
that referenced
this issue
Jul 12, 2024
…2040) Part 1 of #2020 This PR enables local testing of otel metric changes/additions. To use: In tab 1, run `just otel-stream`. In tab 2, run `just otel-dev`. You can pass in any args you'd run `ftl dev` with (e.g. `--recreate`). If you set `--log-level=DEBUG`, you should see `OTEL is enabled, exporting to localhost:4317` in your output. Any signals you trigger to log at this point should appear in the streaming output of tab 1. When you're done, kill the `dev` command and then run `just otel-stop` in any tab other than tab 1.
|
FYI otel-collector is in the docker-compose.yml file, in the "infra" profile. |
deniseli
added a commit
that referenced
this issue
Jul 15, 2024
Part 2 of #2020 This PR adds `just otel-ui`, which wraps the otel collector so you can view signals generated in `ftl dev` through a webapp without separately needing to start the collector. Unfortunately, the latest version of `otel-desktop-viewer` that we can install only supports traces, not metrics or logs. The latest version on their github repo does have metrics/logs support, but can't be installed without cloning the whole repo and building from source code. Instead of doing that, I filed this issue: CtrlSpice/otel-desktop-viewer#146 For now, when we're only debugging/developing traces, we can use this desktop viewer, which is quite nice. When working with metrics/logs, we'll still need to use the collector itself: `just otel-stream`. After the upgrade is complete for `otel-desktop-viewer`, we hopefully won't need the `otel-stream` command anymore. Its terminal output looks like this: ``` $ just otel-ui Installing otel-desktop-viewer... otel-desktop-viewer --grpc 4317 2024-07-15T14:05:07.769-0400 info service/telemetry.go:113 Setting up own telemetry... 2024-07-15T14:05:07.769-0400 info service/telemetry.go:136 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} 2024-07-15T14:05:07.769-0400 info exporter@v0.77.0/exporter.go:286 Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "desktop"} 2024-07-15T14:05:07.769-0400 info exporter@v0.77.0/exporter.go:286 Development component. May change in the future. {"kind": "exporter", "data_type": "logs", "name": "desktop"} 2024-07-15T14:05:07.769-0400 info exporter@v0.77.0/exporter.go:286 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "desktop"} 2024-07-15T14:05:07.769-0400 info service/service.go:141 Starting otel-desktop-viewer... {"Version": "0.1.1", "NumCPU": 12} 2024-07-15T14:05:07.769-0400 info extensions/extensions.go:41 Starting extensions... 2024-07-15T14:05:07.769-0400 info otlpreceiver@v0.77.0/otlp.go:94 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "localhost:4317"} 2024-07-15T14:05:07.771-0400 info otlpreceiver@v0.77.0/otlp.go:112 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "localhost:4318"} 2024-07-15T14:05:07.771-0400 info service/service.go:158 Everything is ready. Begin running and processing data. ^C2024-07-15T14:06:40.238-0400 info otelcol/collector.go:238 Received signal from OS {"signal": "interrupt"} 2024-07-15T14:06:40.238-0400 info service/service.go:167 Starting shutdown... 2024-07-15T14:06:40.238-0400 info extensions/extensions.go:55 Stopping extensions... 2024-07-15T14:06:40.238-0400 info service/service.go:181 Shutdown complete. ``` Screenshot of traces after starting `ftl dev` and calling `echo.echo` via the console: <img width="1333" alt="Screenshot 2024-07-15 at 2 06 31 PM" src="https://github.com/user-attachments/assets/f049caa9-0a64-4286-9648-8429aee466eb"> For posterity, an alternative to using the desktop viewer is the following command to start a terminal ui. However, that doesn't include nearly as much viewable data as the desktop alternative. ``` otel-cli server tui --verbose --protocol grpc --endpoint http://localhost:4317 ```
deniseli
added a commit
that referenced
this issue
Jul 16, 2024
…utput (#2078) Part 3 of #2020 This PR injects newlines between blocks in `just otel-stream`'s docker container output. We should ideally add [color](https://www.unix.com/unix-for-dummies-questions-and-answers/134824-using-sed-change-specific-words-color.html) + bolding to this as well, which can be done by piping to a very similar `sed` command, but moving forward without that for now to save time. Example snippet of the output: ``` Metric #2 Descriptor: -> Name: rpc.server.response.size -> Description: Measures size of RPC response messages (uncompressed). -> Unit: By -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> rpc.grpc.status_code: Int(0) -> rpc.method: Str(Ping) -> rpc.service: Str(xyz.block.ftl.v1.VerbService) -> rpc.system: Str(grpc) StartTimestamp: 2024-07-16 00:02:17.426033 +0000 UTC Timestamp: 2024-07-16 00:02:37.427409 +0000 UTC Count: 1 Sum: 0.000000 Min: 0.000000 Max: 0.000000 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Run otel collector in a docker container to collect local logs/metrics/traces/spans and show the stream as they come in in the terminal tab. This way, we can run ftl dev, trigger the events we plan to log, and see them come through the terminal window in real(ish) time.
Wes's command to start working with:
The text was updated successfully, but these errors were encountered: