WIP: Initial sketch of streaming tail workers #2852

jasnell · 2024-10-07T17:18:02Z

Implements the key pieces of the new streaming tail workers model. Does not yet introduce the changes to make it active. This PR focuses on the core data model and restructuring of parts of the current tail workers impl to accomodate both models. Look at the tests (specifically trace-streaming-test.c++) to get a good feel for how the streaming trace is structured.

Among other changes included here...

Enables tail workers in local dev. A workerd worker can now be configured to push tail events to a tail worker locally. In the current form this only supports the Version 1 tail workers but will soon support the streaming tail workers also.
Adds an autogate that will be used to gate the use of the new streaming tail workers functionality

src/workerd/io/trace-common.h

src/workerd/io/trace-common.c++

src/workerd/io/trace-common.h

src/workerd/server/server.c++

src/workerd/io/trace-common-test.c++

src/workerd/io/trace-common.h

kentonv · 2024-10-08T16:02:20Z

Can we please get a design doc showing what the API looks like from the application developer's perspective? It's hard to figure that out from a PR.

jasnell · 2024-10-08T16:13:36Z

Can we please get a design doc showing what the API looks like from the application developer's perspective...

There's already an internal wiki doc discussing this that has already been shared with you previously. Check your email. Was shared about a month ago and updated yesterday

src/workerd/io/worker-interface.capnp

src/workerd/server/server.c++

src/workerd/io/worker-interface.capnp

src/workerd/io/trace.c++

danlapid · 2024-10-11T17:57:45Z

Overall LGTM

kentonv · 2024-10-11T19:05:23Z

src/workerd/io/worker-interface.capnp

+  # contains general metadata about the pipeline that is being traced.
+  struct Onset {
+    accountId @0 :UInt32;
+    stableId @1 :Text;


Some fields in this struct are internal details which should not be revealed to application code. Why include them here?

Can you please document what the JavaScript API will look like? Your design doc only shows this huge Cap'n Proto definition but doesn't explicitly document how it looks in JavaScript.

Can you please document what the JavaScript API will look like?

This PR is focused mostly on the internal structure and does not currently reflect the JS interface. I do not imagine that the actual JavaScript interfaces will look much different than the current tail workers API defined by api/trace.h except that I'm hoping to be able to do away with everything being a jsg::Object as opposed to just Plain Ole JavaScript objects. The specifics of the JS-facing API will be addressed in a follow up PR.

jasnell · 2024-10-11T19:30:56Z

@danlapid ... thanks for the review! I updated a couple of those. On the naming concern, I'm going to leave the names as they are for now. Those will be trivial to change later if necessary before this goes live.

Adds the ability for a workerd worker configuration to specify a tail worker configuration. This is useful for testing tail worker development locally, which up to now has not been possible.

jasnell · 2024-10-11T23:03:30Z

@fhanau ... updated here to incorporate the new ExecutionModel info

jasnell requested review from byule, irvinebroque, anonrig, mikenomitch and kflansburg October 7, 2024 17:22

jasnell force-pushed the jsnell/stream-trace-workers-part-1 branch 2 times, most recently from 31ff8c6 to 2dcbac7 Compare October 7, 2024 17:50

anonrig reviewed Oct 7, 2024

View reviewed changes

jasnell added the needs-internal-pr label Oct 7, 2024

jasnell force-pushed the jsnell/stream-trace-workers-part-1 branch 2 times, most recently from 794cdc2 to e40d411 Compare October 7, 2024 22:15

jasnell marked this pull request as ready for review October 8, 2024 00:14

jasnell requested review from a team as code owners October 8, 2024 00:14