diff --git a/__tests__/__snapshots__/documentation.js.snap b/__tests__/__snapshots__/documentation.js.snap index 7b551a7f94cd9..54c25636fb6ca 100644 --- a/__tests__/__snapshots__/documentation.js.snap +++ b/__tests__/__snapshots__/documentation.js.snap @@ -277,7 +277,6 @@ Array [ "performance/discover/index.html", "performance/discover/query-builder/index.html", "performance/distributed-tracing/index.html", - "performance/performance-glossary/index.html", "platforms/android/index.html", "platforms/android/migrate/index.html", "platforms/cocoa/dsym/index.html", diff --git a/src/_assets/img/performance/anatomy-of-transaction.png b/src/_assets/img/performance/anatomy-of-transaction.png deleted file mode 100644 index 9b4bce6dd5fb3..0000000000000 Binary files a/src/_assets/img/performance/anatomy-of-transaction.png and /dev/null differ diff --git a/src/_assets/img/performance/span-parent-child-relationship.png b/src/_assets/img/performance/span-parent-child-relationship.png new file mode 100644 index 0000000000000..add2443b57fa9 Binary files /dev/null and b/src/_assets/img/performance/span-parent-child-relationship.png differ diff --git a/src/_assets/img/performance/trace-transactions-spans-concrete-example.png b/src/_assets/img/performance/trace-transactions-spans-concrete-example.png new file mode 100644 index 0000000000000..9676b84100891 Binary files /dev/null and b/src/_assets/img/performance/trace-transactions-spans-concrete-example.png differ diff --git a/src/_assets/img/performance/trace-transactions-spans-generic.png b/src/_assets/img/performance/trace-transactions-spans-generic.png new file mode 100644 index 0000000000000..2d05898975fd6 Binary files /dev/null and b/src/_assets/img/performance/trace-transactions-spans-generic.png differ diff --git a/src/_assets/img/performance/tracing-diagram.png b/src/_assets/img/performance/tracing-diagram.png deleted file mode 100644 index 632b9dc3576d2..0000000000000 Binary files a/src/_assets/img/performance/tracing-diagram.png and /dev/null differ diff --git a/src/collections/_documentation/data-management/advanced-datascrubbing.md b/src/collections/_documentation/data-management/advanced-datascrubbing.md index 13019ee7cf255..fe251420aa776 100644 --- a/src/collections/_documentation/data-management/advanced-datascrubbing.md +++ b/src/collections/_documentation/data-management/advanced-datascrubbing.md @@ -160,7 +160,7 @@ Select known parts of the schema using the following: * `$logentry` matches the `logentry` attribute of an event. * `$thread` matches a single thread instance in `{"threads": {"values": [...]}}` * `$breadcrumb` matches a single breadcrumb in `{"breadcrumbs": {"values": [...]}}` -* `$span` matches a [trace span]({% link _documentation/performance/performance-glossary.md %}#span) +* `$span` matches a [trace span]({% link _documentation/performance/distributed-tracing.md %}#traces-transactions-and-spans) * `$sdk` matches the SDK context in `{"sdk": ...}` ### Escaping Special Characters diff --git a/src/collections/_documentation/performance/distributed-tracing.md b/src/collections/_documentation/performance/distributed-tracing.md index b10cb82cf49e5..98fc813c774f6 100644 --- a/src/collections/_documentation/performance/distributed-tracing.md +++ b/src/collections/_documentation/performance/distributed-tracing.md @@ -12,105 +12,333 @@ Sentry's Performance features are currently in beta. For more details about acce level="warning" %} -Enabling tracing data collection augments your existing error data to capture interactions among your software systems. This lets Sentry tell you valuable metrics about your software health like throughput and latency, as well as expose the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution to diagnose and measure your application health. +Enabling tracing augments your existing error data by capturing interactions among your software systems. With tracing, Sentry tracks your software performance, measuring things like throughput and latency, and can also display the impact of errors across multiple systems. Tracing makes Sentry a more complete monitoring solution, helping you both diagnose problems and measure your application's overall health more easily. Tracing in Sentry provides insights such as: -## Tracing & Distributed Tracing +- What occurred for a specific error event or issue +- The conditions that cause bottlenecks or latency issues in your application +- The endpoints or operations that consume the most time -Applications (for example, web applications) typically consist of interconnected components (also called services). For instance, let us assume a modern web application could be composed of the following components separated by the network boundaries: +## What is Tracing? + +To begin, a note about what tracing is not: Tracing is not profiling. Though the goals of profiling and tracing overlap quite a bit, and though they can both be used to diagnose problems in your application, they differ in terms of what they measure and how the data is recorded. + +A [profiler](https://en.wikipedia.org/wiki/Profiling_(computer_programming)) may measure any number of aspects of an application's operation: the number of instructions executed, the amount of memory being used by various processes, the amount of time a given function call takes, and many more. The resulting profile is a statistical summary of these measurements. + +A [tracing tool](https://en.wikipedia.org/wiki/Tracing_(software)), on the other hand, focuses on _what_ happened (and when), rather than how many times it happened or how long it took. The resulting trace is a log of events which occurred during a program's execution, often across multiple systems. Though traces most often - or, in the case of Sentry's traces, always - include timestamps, allowing durations to be calculated, measuring performance is not their only purpose. They can also show the ways in which interconnected systems interact, and the ways in which problems in one can cause problems in another. + +### Why Tracing? + +Applications typically consist of interconnected components, which are also called services. As an example, let's look at a modern web application, composed of the following components, separated by network boundaries: - Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job +- Backend (REST API) +- Task Queue +- Database Server +- Cron Job Scheduler -Each of these components may be written in a different language on a different platform. Today, all of these components can be instrumented individually with Sentry SDKs to capture error data or crash reports whenever an event occurs in any one of them. +Each of these components may be written in a different language on a different platform. Each can be instrumented individually using a Sentry SDK to capture error data or crash reports, but that instrumentation doesn't provide the full picture, as each piece is considered separately. Tracing allows you to tie all of the data together. -With tracing, we can follow the journey of the API endpoint requests from their source (for example, from the frontend), and instrument code paths as these requests traverse each component of the application. This journey is called a [**trace**]({%- link _documentation/performance/performance-glossary.md -%}#trace). Traces that cross between components, as in our web application example, are typically called **distributed traces**. +In our example web application, tracing means being able to follow a request from the frontend to the backend and back, pulling in data from any background tasks or notification jobs that request creates. Not only does this allow you to correlate Sentry error reports, to see how an error in one service may have propagated to another, but it also allows you to gain stronger insights into which services may be having a negative impact on your application's overall performance. -[{% asset performance/tracing-diagram.png alt="Diagram illustrating how a trace is composed of multiple transactions." %}]({% asset performance/tracing-diagram.png @path %}) +Before learning how to enable tracing in your application, it helps to understand a few key terms and how they relate to one another. -Each [trace]({%- link _documentation/performance/performance-glossary.md -%}#trace) has a marker called a `trace_id`. Trace IDs are pseudorandom fixed-length alphanumeric character sequences. +### Traces, Transactions, and Spans -By collecting traces of your users as they use your applications, you can begin to reveal some insights such as: +A **trace** represents the record of the entire operation you want to measure or track - like page load, an instance of a user completing some action in your application, or a cron job in your backend. When a trace includes work in multiple services, such as those listed above, it's called a **distributed trace**, because the trace is distributed across those services. -- What occurred for a specific error event, or issue -- What conditions are causing bottlenecks or latency issues in the application -- Which endpoints or operations consume the most time +Each trace consists of one or more tree-like structures called **transactions**, the nodes of which are called **spans**. In most cases, each transaction represents a single instance of a service being called, and each span within that transaction represents that service performing a single unit of work, whether calling a function within that service or making a call to a different service. Here's an example trace, broken down into transactions and spans: -### Trace Propagation Model +[{% asset performance/trace-transactions-spans-generic.png alt="Diagram illustrating how a trace is composed of multiple transactions, and each transaction is composed of multiple spans." %}]({% asset performance/trace-transactions-spans-generic.png @path %}) -Let's refer back to our earlier example of the web application consisting of these high-level components: +Because a transaction has a tree structure, top-level spans can themselves be broken down into smaller spans, mirroring the way that one function may call a number of other, smaller functions; this is expressed using the parent-child metaphor, so that every span may be the **parent span** to multiple other **child spans**. Further, since all trees must have a single root, one span in every transaction always represents the transaction itself, with all other spans in the transaction descending from that root span. Here's a zoomed-in view of one of the transactions from the diagram above: -- Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job +[{% asset performance/span-parent-child-relationship.png alt="Diagram illustrating the parent-child relationship between spans within a single transaction." %}]({% asset performance/span-parent-child-relationship.png @path %}) -A trace [propagates]({%- link _documentation/performance/performance-glossary.md -%}#propagation) first from the frontend, then to the backend, and later to the background queue or notification job. Collected [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) from each component are sent back to Sentry asynchronously and independently. Instrumented spans received from one component aren't forwarded to the next component. If a span appears to be missing a parent span, it could be an [**orphan span**]({%- link _documentation/performance/performance-glossary.md -%}#orphan-spans). +To make all of this more concrete, let's consider our example web app again. -## Transactions +### Example: Investigating Slow Page Load -[{% asset performance/anatomy-of-transaction.png alt="Diagram illustrating how a transaction is composed of many spans." %}]({% asset performance/anatomy-of-transaction.png @path %}) +Suppose your web application is slow to load, and you'd like to know why. A lot has to happen for your app to first get to a usable state: multiple requests to your backend, likely some work - including calls to your database or to outside APIs - completed before responses are returned, and processing by the browser to render all of the returned data into something meaningful to the user. So which part of that process is slowing things down? -Traces in Sentry are segmented into [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) called [**transactions**]({%- link _documentation/performance/performance-glossary.md -%}#transaction). When instrumenting the application with tracing, collected spans are grouped within an encompassing top-level span called the **transaction span**. This notion of a transaction is specific to Sentry. +Let's say, in this simplified example, that when a user loads the app in their browser, the following happens in each service: -If you are collecting transactions from more than a single machine, you will likely encounter [**clock skew**]({%- link _documentation/performance/performance-glossary.md -%}#clock-skew). +- _Browser_ + - 1 request each for HTML, CSS, and JavaScript + - 1 rendering task, which sets off 2 requests for JSON data +^ +- _Backend_ + - 3 requests to serve static files (the HTML, CSS, and JS) + - 2 requests for JSON data + - 1 requiring a call to the database + - 1 requiring a call to an external API and work to process the results before returning them to the frontend +^ +- _Database Server_ + - 1 request which requires 2 queries + - 1 query to check authentication + - 1 query to get data + +_Note:_ The external API is not listed precisely because it's external, and you therefore can't see inside of it. -## Sampling Transactions +In this example, the entire page-loading process, including all of the above, is represented by a single **trace**. That trace would consist of the following **transactions**: -**We strongly recommend sampling your transactions.** +- 1 browser transaction (for page load) +- 5 backend transactions (one for each request) +- 1 database server transaction (for the single DB request) + +Each transaction would be broken down into **spans** as follows: -When you enable sampling for transaction events in Sentry, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of 0.25 would result in 250 transactions (25%) being sent to Sentry. +- _Browser Page-load Transaction_: 7 spans + - 1 root span representing the entire page load + - 1 span each (3 total) for the HTML, CSS, and JS requests + - 1 span for the rendering task, which itself contains + - 2 child spans, one for each JSON request -Sampling enables you to collect traces on a subset of your traffic and extrapolate to the total volume. Furthermore, **enabling sampling allows you to control the volume of data you send to Sentry and lets you better manage your costs**. If you don't have a good understanding of what sampling rate to choose, we recommend you start with a low value and gradually increase the sampling rate as you learn more about your traffic patterns and volume. +Let's pause here to make an important point: Some, though not all, of the spans listed here in the browser transaction have a direct correspondence to backend transactions listed earlier. Specifically, each request span in the browser transaction corresponds to a separate request transaction in the backend. In this situation, when a span in one service gives rise to a transaction in a subsequent service, we call the original span a parent span to _both_ the transaction and its root span. In the diagram below, the squggly lines represent this parent-child relationship. -When you have multiple projects collecting transaction events, Sentry utilizes "head-based" sampling to ensure that once a sampling decision has been made at the beginning of the trace (typically the initial transaction), that decision is propagated to each service or project involved in the [trace]({%- link _documentation/performance/performance-glossary.md -%}#trace). If your services have multiple entry points, you should aim to choose a consistent sampling rate for each. Choosing different sampling rates can bias your results. Sentry does not support "tail-based" sampling at this time. +[{% asset performance/trace-transactions-spans-concrete-example.png alt="Diagram illustrating the trace-transaction-span relationship illustrated above, now applied to the example." %}]({% asset performance/trace-transactions-spans-concrete-example.png @path %}) -If you enable Performance collection for a large portion of your traffic, you may exceed your organization's [Quotas and Rate Limits]({%- link _documentation/accounts/quotas/index.md -%}). +In our example, every transaction other than the initial browser page-load transaction is the child of a span in another service, which means that every root span other than the browser transaction root has a parent span (albeit in a different service). -## Viewing Transactions +In a fully-instrumented system (one in which every service has tracing enabled) this pattern will always hold true. The only parentless span will be the root of the initial transaction; every other span will have a parent. Further, parents and children will always live in the same service, except in the case where the child span is the root of a child transaction, in which case the parent span will live in the calling service and the child transaction/child root span will live in the called service. -View transaction events by clicking on the "Transactions" pre-built query in [Discover]({%- link _documentation/performance/discover/index.md -%}), or by using a search condition `event.type:transaction` in a [Discover Query Builder]({%- link _documentation/performance/discover/query-builder.md -%}) view. +Put another way, a fully-instrumented system creates a trace which is itself a connected tree - with each transaction a subtree - and in that tree, the boundaries between subtrees/transactions are precisely the boundaries between services. The diagram above shows one branch of our example's full trace tree. -When you open a transaction event in Discover, you'll see the **span view** at the top of the page. Other information the SDK captured as part of the Transaction event, such as breadcrumbs, will also be displayed. +Now, for the sake of completeness, back to our spans: -[{% asset performance/discover-span.png alt="Discover span showing the map of the transactions (aka minimap) and the black dashed handlebars for adjusting the window selection." %}]({% asset performance/discover-span.png @path %}) +- _Backend HTML/CSS/JS Request Transactions_: 1 span each + - 1 root span representing the entire request (child of a browser span) +^ +- _Backend Request with DB Call Transaction_: 2 spans + - 1 root span representing the entire request (child of a browser span) + - 1 span for querying the database (parent of the database server transaction) +^ +- _Backend Request with API Call Transaction_: 3 spans + - 1 root span representing the entire request (child of a browser span) + - 1 span for the API request (unlike with the DB call, _not_ a parent span, since the API is external) + - 1 span for processing the API data +^ +- _Database Server Request Transaction_: 3 spans + - 1 root span representing the entire request (child of the backend span above) + - 1 span for the authentication query + - 1 span for the query retrieving data + +To wrap up the example: after instrumenting all of your services, you might discover that - for some reason - it's the auth query in your database server that is making things slow, accounting for more than half of the time it takes for your entire page load process to complete. Tracing can't tell you _why_ that's happening, but at least now you know where to look! + +### Further Examples + +This section contains a few more examples of tracing, broken down into transactions and spans. + +#### Measuring a Specific User Action + +If your application involves e-commerce, you likely want to measure the time between a user clicking "Submit Order" and the order confirmation appearing, including tracking the submitting of the charge to the payment processor and the sending of an order confirmation email. That entire process is one trace, and typically you'd have transactions (_T_) and spans (_S_) for: + +- The browser's full process (_T_ and root span _S_) + - XHR request to backend\* (_S_) + - Rendering confirmation screen (_S_) +^ +- Your backend's processing of that request (_T_ and root span _S_) + - Function call to compute total (_S_) + - DB call to store order\* (_S_) + - API call to payment processor (_S_) + - Queuing of email confirmation\* (_S_) +^ +- Your database's work updating the customer's order history (_T_ and root span _S_) + - Individual SQL queries (_S_) +^ +- The queued task of sending the email (_T_ and root span _S_) + - Function call to populate email template (_S_) + - API call to email-sending service (_S_) + +_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). + +#### Monitoring a Background Process + +If your backend periodically polls for data from an external service, processes it, caches it, and then forwards it to an internal service, each instance of this happening is a trace, and you'd typically have transactions (_T_) and spans (_S_) for: + +- The cron job that completes the entire process (_T_ and root span _S_) + - API call to external service (_S_) + - Processing function (_S_) + - Call to caching service\* (_S_) + - API call to internal service\* (_S_) +^ +- The work done in your caching service (_T_ and root span _S_) + - Checking cache for existing data (_S_) + - Storing new data in cache (_S_) +^ +- Your internal service's processing of the request (_T_ and root span _S_) + - Anything that service might do to handle the request (_S_) + +_Note:_ Starred spans represent spans that are the parent of a later transaction (and its root span). + +### Tracing Data Model + +> "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." +> +> -- [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks), The Mythical Man Month (1975) + +While the theory is interesting, ultimately any data structure is defined by the kind of data it contains, and relationships between data structures are defined by how links between them are recorded. Traces, transactions, and spans are no different. + +#### Traces + +Traces are not an entity in and of themselves. Rather, a trace is defined as the collection of all transactions that share a `trace_id` value. + +#### Transactions + +Transactions share most of their properties (start and end time, tags, and so forth) with their root spans, so the same options described below for spans are available in transactions, and setting them in either place is equivalent. + +Transactions also have one additional property not included in spans, called `transaction_name`, which is used in the UI to identify the transaction. Common examples of `transaction_name` values include endpoint paths (like `/store/checkout/` or `api/v2/users/<user_id>/`) for backend request transactions, task names (like `data.cleanup.delete_inactive_users`) for cron job transactions, and URLs (like `https://docs.sentry.io/performance/distributed-tracing/`) for page-load transactions. + +_Note:_ Before the transaction is sent, the `tags` and `data` properties will get merged with data from the global scope. (Global scope data is set either in `Sentry.init()` - for things like `environment` and `release` - or by using `Sentry.configureScope()`, `Sentry.setTag()`, `Sentry.setUser()`, and `Sentry.setExtra()`. See the [Additional Data]({%- link _documentation/enriching-error-data/additional-data.md -%}) docs for more information.) + +#### Spans + +The majority of the data in a transaction resides in the individual spans the transaction contains. Span data includes: + +- `parent_span_id`: ties the span to its parent span +- `op`: short string identifying the type or category of operation the span is measuring +- `start_timestamp`: when the span was opened +- `end_timestamp`: when the span was closed +- `description`: longer description of the span's operation, which uniquely identifies the span but is consistent across instances of the span (optional) +- `status`: short code indicating operation's status (optional) +- `tags`: key-value pairs holding additional data about the span (optional) +- `data`: arbitrarily-structured additional data about the span (optional) + +An example use of the `op` and `description` properties together is `op: sql.query` and `description: SELECT * FROM users WHERE last_active < %s`. The `status` property is often used to indicate the success or failure of the span's operation, or for a response code in the case of HTTP requests. Finally, `tags` and `data` allow you to attach further contextual information to the span, such as `function: middleware.auth.is_authenticated` for a function call or `request: {url: ..., headers: ... , body: ...}` for an HTTP request. + +### Further Information + +A few more important points about traces, transactions, and spans, and the way they relate to one another: + +#### Trace Duration + +Because a trace just is a collection of transactions, traces don't have their own start and end times. Instead, a trace begins when its earliest transaction starts, and ends when its latest transaction ends. As a result, you can't explicitly start or end a trace directly. Instead, you create a trace by creating the first transaction in that trace, and you complete a trace by completing all of transactions it contains. + +#### Async Transactions + +Because of the possibility of asynchronous processes, child transactions may outlive the transactions containing their parent spans, sometimes by many orders of magnitude. For example, if a backend API call sets off a long-running processing task and then immediately returns a response, the backend transaction will finish (and its data will be sent to Sentry) long before the async task transaction does. Asynchronicity also means that the order in which transactions are sent to (and received by) Sentry does not in any way depend on the order in which they were created. (By contrast, order of receipt for transactions in the same trace _is_ correlated with order of completion, though because of factors like the variability of transmission times, the correlation is far from perfect.) + +#### Orphan Transactions + +In theory, in a fully instrumented system, each trace should contain only one transaction and one span (the transaction's root) without a parent, namely the transaction in the originating service. However, in practice, you may not have tracing enabled in every one of your services, or an instrumented service may fail to report a transaction due to network disruption or other unforeseen circumstances. When this happens, you may see gaps in your trace hierarchy. Specifically, you may see transactions partway through the trace whose parent spans haven't been recorded as part of any known transactions. Such non-originating, parentless transactions are called **orphan transactions**. + +#### Nested Spans + +Though our examples above had four levels in their hierarchy (trace, transaction, span, child span) there's no set limit to how deep the nesting of spans can go. There are, however, practical limits: transaction payloads sent to Sentry have a maximum allowed size, and as with any kind of logging, there's a balance to be struck between your data's granularity and its usability. + +#### Zero-duration Spans -### Using the Span View +It's possible for a span to have equal start and end times, and therefore be recorded as taking no time. This can occur either because the span is being used as a marker (such as is done in [the browser's Performance API](https://developer.mozilla.org/en-US/docs/Web/API/Performance/mark)) or because the amount of time the operation took is less than the measurement resolution (which will vary by service). -Note that [traces]({%- link _documentation/performance/performance-glossary.md -%}#trace) are segmented into [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) called [transactions]({%- link _documentation/performance/performance-glossary.md -%}#transaction). The span view enables you to examine the waterfall graph (or hierarchy) of the instrumented transaction. +#### Clock Skew + +If you are collecting transactions from multiple machines, you will likely encounter **clock skew**, where timestamps in one transaction don't align with timestamps in another. For example, if your backend makes a database call, the backend transaction logically should start before the database transaction does. But if the system time on each machine (those hosting your backend and database, respectively) isn't synced to a common standard, it's possible that won't be the case. It's also possible for the ordering to be correct, but for the two recorded timeframes to not line up in a way that accurately reflects what actually happened. To reduce this possibility, we recommend using Network Time Protocol (NTP) or your cloud provider's clock synchronization services. + +#### How Data is Sent + +Individual spans aren't sent to Sentry; rather, the entire transaction is sent as one unit. This means that no span data will be recorded by Sentry's servers until the transaction to which they belong is closed and dispatched. The converse is not true, however - though spans can't be sent without a transaction, transactions _are_ still valid, and will be sent, even if the only span they contain is their root span. + +## Data Sampling + +When you enable sampling in your tracing setup, you choose a percentage of collected transactions to send to Sentry. For example, if you had an endpoint that received 1000 requests per minute, a sampling rate of `0.25` would result in approximately 250 transactions (25%) being sent to Sentry each minute. (The number is approximate because each request is either tracked or not, independently and pseudorandomly, with a 25% probability. So in the same way that 100 fair coins, when flipped, result in approximately 50 heads, the SDK will "decide" to collect a trace in approximately 250 cases.) Because you know the sampling percentage, you can then extrapolate your total traffic volume. + +When collecting traces, we **strongly recommend** sampling your data, for two reasons. First, though capturing a single trace involves minimal overhead, capturing traces for every single page load, or every single API request, has the potential to add an undesirable amount of load to your system. Second, by enabling sampling you'll more easily prevent yourself from exceeding your organization's [event quota]({%- link _documentation/accounts/quotas/index.md -%}), which will help you manage costs. + +When choosing a sampling rate, the goal is to not collect _too_ much data (given the reasons above) but also to collect enough data that you are able to draw meaningful conclusions. If you're not sure what rate to choose, we recommend starting with a low value and gradually increasing it as you learn more about your traffic patterns and volume, until you've found a rate which lets you balance performance and cost concerns with data accuracy. + +### Consistency Within a Trace + +For traces that involve multiple transactions, Sentry uses a "head-based" approach: a sampling decision is made in the originating service, and then that decision is passed to all subsequent services. To see how this works, let's return to our webapp example above. Consider two users, A and B, who are both loading your app in their respective browsers. When A loads the app, the SDK pseudorandomly "decides" to collect a trace, whereas when B loads the app, the SDK "decides" not to. When each browser makes requests to your backend, it includes in those requests the "yes, please collect transactions" or the "no, don't collect transactions this time" decision in the headers. + +When your backend processes the requests from A's browser, it sees the "yes" decision, collects transaction and span data, and sends it to Sentry. Further, it includes the "yes" decision in any requests it makes to subsequent services (like your database server), which similarly collect data, send it to Sentry, and pass the decision along to any services they call. Through this process, all of the relevant transactions in A's trace are collected and sent to Sentry. + +On the other hand, when your backend processes the requests from B's browser, it sees the "no" decision, and as a result it does _not_ collect and send transaction and span data to Sentry. It does, however, do the same thing it does in A's case in terms of propagating the decision to subsequent services, telling them not to collect or send data either. They then in turn tell any services they call not to send data, and in this way no transactions from B's trace are collected. + +Put simply: as a result of this head-based approach, where the decision is made once in the originating service and passed to all subsequent services, either all of the transactions for a given trace are collected, or none are, so there shouldn't be any incomplete traces. + +### Consistency Between Traces + +If you enable tracing in services with multiple entry points, we recommend choosing similar sampling rates, to avoid biasing your data. For example, suppose the backend of our on-going web app example also serves as a public API. In that case, some traces would start with a page-load transaction in the web app, and likely include multiple backend transactions, while other traces (those representing folks hitting the public API) would begin with a single backend request transaction, which would be the only backend transaction in the trace. Choosing a very different sampling rate for your web app from that chosen for your backend would lead to one of those scenarios being oversampled compared to the other, compromising the accuracy of your overall data. + +## Viewing Trace Data + +You can see a list of transaction events by clicking on the "Transactions" pre-built query in [Discover]({%- link _documentation/performance/discover/index.md -%}), or by using a search condition `event.type:transaction` in the [Discover Query Builder]({%- link _documentation/performance/discover/query-builder.md -%}) view. + +### Transaction List View + +The results of either of the above queries are presented in a list view, where each entry represents a group of one or more transactions. Data about each group is displayed in table form, and comes in two flavors: value-based (such as transaction name), and aggregate (such as average duration). The choice of which kinds of data to display is configurable, and can be changed by clicking 'Edit Columns' at the top right of the table. Bear in mind that adding or removing any value-based columns may affect the way the results are grouped. + +This view also includes a timeseries graph, aggregating all results of the query, as well as a summary of the most common tags associated with those results (either via your Sentry instance's [global context]({%- link _documentation/enriching-error-data/additional-data.md -%}) or via each transaction's root span). From this view, you can also filter the transactions list, either by restricting the time window or by adding attributes to the query (or both!). + +_Note:_ Currently, only transaction data - the transaction name and any attributes the transaction inherits from its root span - is searchable. Data contained in spans other than the root span is not indexed and therefore cannot be searched. + +Full documentation of the transaction list view (which is just a special case of the Discover Query Builder) can be found [here]({%- link _documentation/performance/discover/query-builder.md -%}). + +#### Performance Metrics + +A number of the metrics available as column choices lend themselves well to analyzing your application's performance. + +_Transaction Duration Metrics_ + +The following functions aggregate transaction durations: + +- average +- various percentiles (by default, the pre-built Transactions query shows the 75th and 95th percentiles, but there are many other options, including a custom percentile) +- maximum + +One use case for tracking these statistics is to help you identify transactions that are slower than your organization's target SLAs. + +A word of caution when looking at averages and percentiles: In most cases, you'll want to set up tracing so that only [a fraction](#data-sampling) of possible traces are actually sent to Sentry, to avoid overwhelming your system. Further, you may want to filter your transaction data by date or other factors, or you may be tracing a relatively uncommon operation. For all of these reasons, you may end up with average and percentile data that is directionally correct, but not accurate. (To use the most extreme case as an example, if only a single transaction matches your filters, you can still compute an "average" duration, even though that's clearly not what is usually meant by "average.") + +The problem of small sample size (and the resulting inability to be usefully accurate) will happen more often for some metrics than others, and sample size will also vary by row. For example, it takes less data to calculate a meaningful average than it does to calculate an equally meaningful 95th percentile. Further, a row representing requests to `/settings/my-awesome-org/` will likely contain many times as many transactions as one representing requests to `/settings/my-awesome-org/projects/best-project-ever/`. + +_Transaction Frequency Metrics_ + +The following functions aggregate transaction counts and the rate at which transactions are recorded: + +- count +- count unique values (for a given field) +- average requests (transactions) per second +- average requests (transactions) per minute + +Each of these functions is calculated with respect to the collection of transactions within the given row, which means the numbers will change as you filter your data or change the time window. Also, if you have set up your SDK to [sample your data](#data-sampling), remember that only the transactions that are sent to Sentry are counted. So if a row containing transactions representing requests to a given endpoint is calculated to be receiving 5 requests per second, and you've got a 25% sampling rate enabled, in reality you're getting approximately 20 requests to that endpoint each second. (20 because you're collecting 25% - or 1/4 - of your data, so your real volume is 4 times what you're seeing in Sentry.) + +### Transaction Detail View + +When you open a transaction event in Discover (by clicking on the icon at the left side of the row), you'll see the **span view** at the top of the page. Other information the SDK captured as part of the transaction event (such as the transaction's tags and automatically collected breadcrumbs) is displayed underneath and to the right of the span view. + +[{% asset performance/discover-span.png alt="Discover span showing the map of the transactions (aka minimap) and the black dashed handlebars for adjusting the window selection." %}]({% asset performance/discover-span.png @path %}) -The span view is a split view where the left-hand side is the tree view displaying the parent-child relationship of the spans, and the right-hand side displays spans represented as colored rectangles. Within the tree view (left-hand side), Sentry identifies spans by their **operation name** and their **description**. If you don't provide the description, Sentry uses the span id as the fallback. +#### Using the Span View -At the top of the span view is a minimap, which is a "map" of the transaction. It helps orient you to the specific portion of the transaction that you're viewing. +The span view is a split view where the left-hand side shows the transaction's span tree, and the right-hand side represents each span as a colored rectangle. Within the tree view, Sentry identifies spans by their `op` and `description` values. If a span doesn't have a description, Sentry uses the span's id as a fallback. The first span listed is always the transaction's root span, from which all other spans in the transaction descend. -The first top-level span is the transaction span, which encompasses all other spans within the transaction. +At the top of the span view is a minimap, which shows which specific portion of the transaction you're viewing. -**Zooming In on a Transaction** +_Zooming In on a Transaction_ -As displayed in the Discover span screenshot above, you can click and drag your mouse cursor across the minimap (top of the span view). You can also adjust the window selection by dragging the handlebars (black dashed lines). +As shown in the Discover span screenshot above, you can click and drag your mouse cursor across the minimap (top of the span view). You can also adjust the window selection by dragging the handlebars (black dashed lines). -**Missing Instrumentation** +_Missing Instrumentation_ -Sentry may indicate that gaps between spans are "Missing Instrumentation." This message could mean that the SDK was unable to capture or instrument any spans within this gap automatically. It may require you to instrument the gap [manually](#setting-up-tracing). +Sentry may indicate that gaps between spans are "Missing Instrumentation." This means that there is time in the transaction that isn't accounted for by any of the transaction's spans, and likely means that you need to manually instrument that part of your process. -**Viewing Span Details** +_Viewing Span Details_ -Click on a row to expand the details of the span. From here, you can see all attached properties, such as **tags** and **data**. +Clicking on a row in the span view expands the details of that span. From here, you can see all attached properties, such as tags and data. [{% asset performance/span-detail-view.png alt="Span detail view shows the span id, trace id, parent span id, and other data such as tags." %}]({% asset performance/span-detail-view.png @path %}) -**Search by Trace ID** +_Search by Trace ID_ -You can search using `trace id` by expanding any of the span details and click on "Search by Trace". +You can search for all of the transactions in a given trace by expanding any of the span details and clicking on "Search by Trace". -You need **project permissions for each project** to see all the transactions within a trace. Each transaction in a trace is likely a part of a different project. If you don't have project permissions, some transactions may not display as part of a trace. +_Note:_ On the [Team plan](https://sentry.io/pricing/), results will only be shown for one project at a time. Further, each transaction belongs to a specific project, and you will only be able to see transactions belonging to projects you have permission to view. Therefore, you may not see all transactions in a given trace in your results list. -**Traversing to Child Transactions** +_Traversing to Child Transactions_ -Child transactions are shown based on your project permissions -- which are necessary to viewing transaction events. To check project permissions, navigate to **Settings >> [your organization] >> [your project] >> General**. +Some spans within a transaction may be the parent of another transaction. When you expand the span details for such spans, you'll see the "View Child" button, which, when clicked, will lead to the child transaction's details view. -Some spans within a transaction may be the parent of another transaction. If you expand the span details, you may see the "View Child" button, which, when clicked, will lead to another transaction's details view. +_Note:_ Traversing between transactions in this way is only available on the [Business plan](https://sentry.io/pricing/). Further, each transaction belongs to a specific project, and you will only be able to see the "View Child" button if the child transaction belongs to a project you have permission to view. ## Setting Up Tracing @@ -142,16 +370,16 @@ sentry_sdk.init( ) ``` -**Automatic Instrumentation** +#### Automatic Instrumentation -Many integrations for popular frameworks automatically capture traces. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: +Many integrations for popular frameworks automatically capture transactions. If you already have any of the following frameworks set up for Sentry error reporting, you will start to see traces immediately: - All WSGI-based web frameworks (Django, Flask, Pyramid, Falcon, Bottle) - Celery - AIOHTTP web apps - Redis Queue (RQ) -[Spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations within a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction): +Spans are instrumented for the following operations within a transaction: - Database queries that use SQLAlchemy or the Django ORM - HTTP requests made with `requests` or the `stdlib` @@ -169,9 +397,9 @@ sentry_sdk.init( ) ``` -**Manual Instrumentation** +#### Manual Instrumentation -To manually instrument certain regions of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture them. +To manually instrument certain regions of your code, you can create a transaction to capture them. The following example creates a transaction for a scope that contains an expensive operation (for example, `process_item`), and sends the result to Sentry: @@ -186,9 +414,9 @@ while True: process_item(item) ``` -**Adding More Spans to the Transaction** +#### Adding More Spans to the Transaction -The next example contains the implementation of the hypothetical `process_item` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open `transaction` and add all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations to the `transaction`. Keep in mind that each individual span also needs to be manually finished; otherwise, spans will not show up in the `transaction`. +The next example contains the implementation of the hypothetical `process_item` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add all newly created spans as child operations to that transaction. Keep in mind that each individual span also needs to be manually finished; otherwise, spans will not show up in the transaction. You can choose the value of `op` and `description`. @@ -213,7 +441,7 @@ $ npm install @sentry/browser $ npm install @sentry/apm ``` -**Sending Traces/Transactions/Spans** +#### Sending Traces/Transactions/Spans The `Tracing` integration resides in the `@sentry/apm` package. You can add it to your `Sentry.init` call: @@ -232,7 +460,7 @@ Sentry.init({ To send traces, you will need to set the `tracesSampleRate` to a nonzero value. The configuration above will capture 25% of your transactions. -You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) for every page load. Within that transaction, [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations: +You can pass many different options to the Tracing integration (as an object of the form `{optionName: value}`), but it comes with reasonable defaults out of the box. It will automatically capture a transaction for every page load. Within that transaction, spans are instrumented for the following operations: - XHR/fetch requests - If available: Browser Resources (Scripts, CSS, Images ...) @@ -252,9 +480,9 @@ The default value of `tracingOrigins` is `['localhost', /^\//]`. The JavaScript *NOTE:* You need to make sure your web server CORS is configured to allow the `sentry-trace` header. The configuration might look like `"Access-Control-Allow-Headers: sentry-trace"`, but this depends a lot on your set up. If you do not whitelist the `sentry-trace` header, the request might be blocked. -**Using Tracing Integration for Manual Instrumentation** +#### Using Tracing Integration for Manual Instrumentation -The tracing integration will create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) on page load by default; all [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime beyond 500ms, you can do so by adding more spans to the transaction. The following is an example of how you could profile a React component: +The tracing integration will create a transaction on page load by default; all spans that are created will be attached to it. Also, the integration will finish the transaction after the default timeout of 500ms of inactivity. The page is considered inactive if there are no pending XHR/fetch requests. If you want to extend the transaction's lifetime, you can do so by adding more spans to it. The following is an example of how you could manually instrument a React component: ```javascript // This line starts an activity (and creates a span). @@ -305,15 +533,15 @@ shopCheckout() { } ``` -This example will send a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) `shopCheckout` to Sentry, containing all outgoing requests that happen in `validateShoppingCartOnServer` as spans. The transaction will also contain a `task` span that measures how long `processAndValidateShoppingCart` took. Finally, the call to `ApmIntegrations.Tracing.finshIdleTransaction()` will finish the [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) and send it to Sentry. Calling this is optional; if it is not called, the integration will automatically send the [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) itself after the defined `idleTimeout` (default 500ms). +This example will send a transaction `shopCheckout` to Sentry, containing all outgoing requests that happen in `validateShoppingCartOnServer` as spans. The transaction will also contain a `task` span that measures how long `processAndValidateShoppingCart` took. Finally, the call to `ApmIntegrations.Tracing.finishIdleTransaction()` will finish the transaction and send it to Sentry. Calling this is optional; if it is not called, the integration will automatically send the transaction itself after the defined `idleTimeout` (default 500ms). -**What is an activity?** +#### What is an activity? The concept of an activity only exists in the `Tracing` integration in JavaScript Browser. It's a helper that tells the integration how long it should keep the `IdleTransaction` alive. An activity can be pushed and popped. Once all activities of an `IdleTransaction` have been popped, the `IdleTransaction` will be sent to Sentry. -**Manual Transactions** +#### Manual Transactions -To manually instrument a certain region of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture it. +To manually instrument a certain region of your code, you can create a transaction to capture it. This is valid for both JavaScript Browser and Node and works independent of the `Tracing` integration. The following example creates a transaction for a part of the code that contains an expensive operation (for example, `processItem`), and sends the result to Sentry: @@ -331,9 +559,9 @@ processItem(item).then(() => { }) ``` -**Adding Additional Spans to the Transaction** +#### Adding Additional Spans to the Transaction -The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. +The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. ```javascript function processItem(item, transaction) { @@ -365,7 +593,7 @@ $ npm install @sentry/node $ npm install @sentry/apm ``` -**Sending Traces** +#### Sending Traces To send traces, set the `tracesSampleRate` to a nonzero value. The following configuration will capture 25% of all your transactions: @@ -381,7 +609,7 @@ Sentry.init({ }); ``` -**Automatic Instrumentation** +#### Automatic Instrumentation It’s possible to add tracing to all popular frameworks; however, we provide pre-written handlers only for Express.js. @@ -408,11 +636,11 @@ app.use(Sentry.Handlers.errorHandler()); app.listen(3000); ``` -[Spans]({%- link _documentation/performance/performance-glossary.md -%}#span) are instrumented for the following operations within a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction): +Spans are instrumented for the following operations within a transaction: - HTTP requests made with `request` - `get` calls using native `http` and `https` modules -- Middlewares (Express.js only) +- Middleware (Express.js only) To enable this: @@ -434,9 +662,9 @@ Sentry.init({ }); ``` -**Manual Transactions** +#### Manual Transactions -To manually instrument a certain region of your code, you can create a [transaction]({%- link _documentation/performance/performance-glossary.md -%}#transaction) to capture it. +To manually instrument a certain region of your code, you can create a transaction to capture it. The following example creates a transaction for a part of the code that contains an expensive operation (for example, `processItem`), and sends the result to Sentry: @@ -456,9 +684,9 @@ app.use(function processItems(req, res, next) { }); ``` -**Adding Additional Spans to the Transaction** +#### Adding Additional Spans to the Transaction -The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created [spans]({%- link _documentation/performance/performance-glossary.md -%}#span) as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. +The next example contains the implementation of the hypothetical `processItem ` function called from the code snippet in the previous section. Our SDK can determine if there is currently an open transaction and add to it all newly created spans as child operations. Keep in mind that each individual span needs to be manually finished; otherwise, that span will not show up in the transaction. ```javascript function processItem(item, transaction) { diff --git a/src/collections/_documentation/performance/performance-glossary.md b/src/collections/_documentation/performance/performance-glossary.md deleted file mode 100644 index 773a98b84f818..0000000000000 --- a/src/collections/_documentation/performance/performance-glossary.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -title: Performance Glossary -sidebar_order: 5 ---- - -## Clock Skew - -If you are collecting transactions from multiple machines, you will likely encounter **clock skew**. When collecting and comparing transaction events that are gathered from different devices, you may observe timestamps that do not align or events in a trace that don't share a time history. Sentry does not attempt to correct the timestamps in your events. The timestamps displayed in Sentry for transactions and each span retain the original values reported by your application/hosting environment. - -While you can reduce clock skew by utilizing Network Time Protocol (NTP) or your cloud provider's clock synchronization services, you may still notice small drifts in your data, as synchronizing clocks on small intervals is challenging. - -## Performance Metrics - -Some aggregate functions can be applied to transaction events to help you better understand the performance characteristics of your applications. - -### Duration Percentiles - -You can aggregate transaction durations using the following aggregate functions: -- average -- 75%, 95%, and 99% duration percentiles - -One use-case for using these statistics is to help you identify transactions that are slower than your organization's target SLAs. - -Transaction events are sampled. As a result, the percentiles presented within Sentry represent only the data received. Because of sampling, data ranges, filters, or low volume of transactions, we will often have a case where data is directionally correct, but not accurate. For example, the average of one number is that number, but that’s not usually what you may expect to see. - -This inability to be usefully accurate will happen more often in some columns than others. We can calculate an average with less data than a 95th percentile, but it’ll also vary by row. An index page will always get more traffic than `/settings/country/belgium/tax`. - -### Requests Per Minute (RPM) - -Requests Per Minute is a way to measure throughput. It is the average of all requests bucketed by the minute for the current time window and query string. - -## Span - -The **span** is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. It may represent the processing of an HTTP request by a server, the querying of a database, or an HTTP call to an external service. - -When you initialize and load a Sentry SDK that supports tracing, it will automatically instrument parts of your running application, such as function calls and network operations. Instrumenting a database call could entail these three steps or subroutines: - -1. Calling a function to compose a database query -2. Sending a query to a database -3. Receiving and transforming results - -Each of these subroutines takes some time to perform (no matter how fast they could run). Sentry records these subroutines as a unit of work in the form of **spans**. Sentry can also attach useful information to spans, such as tags, additional contextual data, and a status to indicate if the subroutine failed or not. - -Spans can have descendant spans (or child spans). In our earlier example, the three subroutines can be spans within a larger encompassing span (the database call). - -Any instrumented span is part of a trace (identified by its trace id, `trace_id`), and each span has its own marker called the `span_id`, a fixed length of alphanumeric characters. - -An example of a span that describes a call to an external service: - -```javascript -// updateJiraTask -{ - "trace_id": "a8233eb845154a2cae4712b28375d08d", - "parent_span_id": "ba99c37bfb30f860", - "span_id": "92ce7ea47d5b8a45", - "start_timestamp": 1583868931.940406, - "same_process_as_parent": true, - "description": "updateJiraTask", - "tags": { "status": "ok" }, - "timestamp": 1583868932.060615, - "op": "jira", - "data": {} -} - -// sql.update - a child span of updateJiraTask -{ - "trace_id": "a8233eb845154a2cae4712b28375d08d", - "parent_span_id": "92ce7ea47d5b8a45", - "span_id": "a88e33dab5b745b9", - "start_timestamp": 1583868931.953386, - "same_process_as_parent": true, - "description": "sql.update - SELECT * FROM jira_table", - "tags": { "status": "ok" }, - "timestamp": 1583868932.03692, - "op": "db", - "data": {} -} -``` - -### Hierarchy - -Each span can be connected to other spans using a parent-child hierarchy. - -For example, a span called `updateJiraTask` would have child spans like `Check permissions`, `Update Task`, `sql.update`, `send.webhooks`. - -Each sub-task span would have `parent_span_id` equal to the id of the `updateJiraTask` span. - -### Orphan Spans - -Orphan spans are spans that lack visible parent spans within Sentry's database. This could imply a bug in the Sentry application. If you're confident about this, please contact support: [support@sentry.io](mailto:support@sentry.io) - -Another reason could include that instrumented spans may not make it back to Sentry for processing. As an example, let us consider a web application consisting of: - -- Frontend (Single-Page Application) -- Backend -- Background Queue -- Notification Job - -Instrumented spans in each component are sent back to Sentry for processing. However, a server in the backend component could suffer from network connectivity loss, and the spans may not have made it back to Sentry. This would indicate a "hole" within this specific trace. The collected spans within the Background Queue would be descendants of a span (for example, a queue service call) within the Backend component, and thus the top-level span of the collective Background Queue spans is referred to as an orphan span. - -### Properties - -Each span has a unique identifier kept in the `span_id` attribute. - -Start time and end time are stored in the `start_timestamp` and `timestamp` attributes, respectively. The format is an RFC 3339 / ISO 8601-compatible string or a numeric value with the number of seconds since the UNIX Epoch. - -The name of the operation is stored in the `op` property. Example `op` values are `http`, `sql.query`, `redis.command`. - -Additional data about the operation can be stored in the `data` and `tags` attributes. - -## Trace - -A trace is a collection of transactions that have the same `trace_id` attribute. Traces consist of all transactions generated by the distributed system during the processing of the input data. Traces are not explicitly started or finished. The start and end of a trace are defined by the transactions and spans participating in the trace. - -Traces that start with the same set of API endpoint calls can potentially differ in terms of what code paths were executed. For example, an initial trace may result in a cache miss at the Cache Service, and a subsequent trace may include more information instrumented at the Cache Service. Although these two traces are similar, they are not identical. - -### Propagation - -Processing of the input data in the distributed environment means that an arbitrary number of the systems/microservices can be involved. - -To connect all the spans that were created during the processing of the input data, we need to be able to propagate the basic identifier. By forwarding `trace_id` and `parent_span_id` as data travels through various applications in a microservice ecosystem, we can reconstruct data flows later. - -Trace metadata is propagated through HTTP headers that are included in the HTTP request to the external system. - -## Transaction - -Transaction is an event attribute that provides initial context to the error report. This makes it possible for the error to be linked to a specific part of the system. An example of the transaction name can be “/users/” which is the name for the endpoint that received the request that failed during the processing. - -With tracing, the transaction is a parent span to all spans created during the processing of the request in the single server. - -Once the request is processed, the transaction is sent together with all child spans to Sentry.