Add support for tracing asynchronous code #87
Conversation
|
Hi, Sorry for the big code dump. I wanted to be sure that this approach to adding async support actually worked to begin with. Please let me know if there is anything i can do to make this acceptable for merging. |
|
Thanks for the PR! Taking a look this week. |
There was a problem hiding this comment.
Thanks for wading in and implementing this, we've wanted it for a while. :)
Couple issues below and one optional thing but otherwise looking good from my perspective. Note I'm not super experienced with asyncio patterns so it's possible I've missed something, but the synchronous stuff passes my smoke test script and as long as we don't break any existing functionality I'm happy to iterate and work out any asyncio bugs that do show up.
In TestBeeline.test_treaded_trace there are some assertions run in threads. Assertion failures from these would only appear in the log, but not fail the test. This change re-raises any exceptions in the main thread so that they will cause test failures.
This moves the trace state from attributes on the threading.local object to a single Trace object. The new trace object is made available as the _trace attribute on SynchronousTracer objects. Behavior is unmodified except in some corner cases: some additional operations will log a warning if they are performed while no trace is started, like updating trace fields.
This change moves all code except the Trace object access implementation up to the generic Tracer class.
This change adds AsyncioTracer, which is chosen instead of SynchronousTracer as the tracer implementation if beeline is initialized from within an asyncio event loop. AsyncioTracer stores its trace state in a context variable. This requires the contextvars module, which was added in Python 3.7. AsyncioTracer also sets up the task factory for the event loop to hook in a copying of the trace state for each new task. This ensures that trace state behaves as expected for parallel running tasks. The CI jobs have been updated to handle that some of the new code uses async/await syntax that wasn't available in Python 2.7.
niligulmohar
force-pushed
the
asyncio-r2
branch
from
ca538f7 to
6698857
Compare
|
Rebased on top of v2.10.1 . |
|
awesome! |
This change adds
AsyncioTracer, which is chosen instead ofSynchronousTraceras the tracer implementation if beeline isinitialized from within an
asyncioevent loop.AsyncioTracerstores its trace state in a context variable insteadof in thread local storage. This requires the
contextvarsmodule,which was added in Python 3.7. Otherwise the implementation is shared
with
SynchronousTracer. The trace state implementation is moved intothe
Traceclass to support both implementations.Context variable assignments are unique per
asynciotask, allowingseparate trace state per task. New tasks automatically receive the
same context variable bindings when they are created, which is not
what you want for the trace state, since they need to be able to
diverge between tasks. To fix this,
AsyncioTracersets up a taskfactory on the event loop that creates a copy of the trace state for
each task. The task factory chain-calls to the previous one, if one
happens to already be configured.
A new
beeline.untracedfunction decorator is added, to avoidautomatically sharing trace state for
asynciotasks that are meantto be unrelated.
The async code requires syntax that is incompatible with Python 2.7,
and modules of which all aren't available until Python 3.7. The code
itself checks for the necessary features by handling
ImportError.The test suite/CI jobs now have separate configurations depending on
the Python version.