- User Manual
The EaseAgent configuration information can be divided into two categories, one is the global configuration and the other is the plugin configuration.
Global configuration include dedicated parameters for controlling metrics and tracing collection behavior via agent.properties. These parameters include:
- Data reporting frequency
- Data reporting output type
- Kafka topic of data reporting
- Data collecting and reporting switch
- Queue depth in process for high throughput
Plugin level configuration provides more granular control and customizable configuration.
The agent can consume configuration from one or more of the following sources (ordered from highest to lowest priority):
- system properties
- environment variables
- the configuration file
You may extract default configuration from the JAR file or create new properties from a blank file.
$ jar xf easeagent.jar agent.properties easeagent-log4j2.xml
Run the user application with EaseAgent
$ export EASE_AGENT_PATH=[Replace with agent path]
$ java "-javaagent:${EASE_AGENT_PATH}/easeagent.jar" -Deaseagent.config.path=${EASE_AGENT_PATH}/agent.properties -jar user-app.jar
or
$ export EASE_AGENT_PATH=[Replace with agent path]
$ export EASEAGENT_CONFIG_PATH=${EASE_AGENT_PATH}/agent.properties
$ java "-javaagent:${EASE_AGENT_PATH}/easeagent.jar" -jar user-app.jar
| System property | Environment variable | Configuration File Key | Description |
|---|---|---|---|
easeagent.name |
EASEAGENT_NAME |
name |
Specify logical service name. |
easeagent.system |
EASEAGENT_SYSTEM |
system |
Specify logical service system. |
EaseAgent opens port 9900 by default to receive configuration change notifications and Prometheus requests.
| System property | Environment variable | DefaultValue | Description |
|---|---|---|---|
easeagent.server.enabled |
EASEAGENT_SERVER_ENABLED |
true | Enable Internal HTTP Server. false can disable it. EaseAgent will no longer accept any HTTP requests (Prometheus、Health Check、Readiness Check、Agent Info) when the Internal HTTP Server is disabled. User can add VM parameter:-Deaseagent.server.enabled=[true or false] to override. |
easeagent.server.port |
EASEAGENT_SERVER_PORT |
9900 | Internal HTTP Server port. User can add VM parameter:-Deaseagent.server.port=[new port] to override. |
Tracing and metric data can be output to kafka server.
| Key | Default Value | Description |
|---|---|---|
reporter.outputServer.bootstrapServer |
127.0.0.1:9092 | Kafka server host and port. Tracing and metric data will be output to kafka. |
reporter.outputServer.timeout |
10000 | Connect timeout. Time Unit: millisecond. |
Global configuration for tracing output
| Key | Default Value | Description |
|---|---|---|
reporter.tracing.sender.appendType |
console | console : output tracing to console; kafka : output tracing to kafka output server; http: send data to http server(zipkin) server |
reporter.tracing.sender.url |
http://localhost:9411/api/v2/spans | Zipkin(HTTP) server url, only available when reporter.tracing.sender.appendType=http |
reporter.tracing.sender.topic |
log-tracing | kafka topic, only available when reporter.tracing.sender.appendType=kafka |
Following tracing sender configuration items:
| Key | Default Value | Description |
|---|---|---|
reporter.tracings.output.enabled |
true | true: enable output tracing data;false: disable all tracing data output |
reporter.tracings.output.messageMaxBytes |
999900 | Maximum bytes sendable per message including encoding overhead. |
reporter.tracings.output.queuedMaxSpans |
1000 | Maximum backlog of spans reported before sent. |
reporter.tracings.output.queuedMaxSize |
1000000 | Maximum backlog of span bytes reported before sent. |
reporter.tracings.output.messageTimeout |
1000 | Spans are bundled into messages, up to messageMaxBytes. This timeout starts when the first unsent span is reported, which ensures that spans are not stuck in an incomplete message. |
Configuration for access log output are similar to tracing:
| Key | Default Value | Description |
|---|---|---|
reporter.log.sender.appendType |
console | console : output log to console; kafka : output log to kafka output server; http: send data to http server(zipkin) server |
reporter.log.sender.url |
/application-log | HTTP server url, only available when reporter.log.sender.appendType=http |
reporter.log.sender.topic |
applicaton-log | kafka topic, only available when reporter.log.sender.appendType=kafka |
| Key | Default Value | Description |
|---|---|---|
reporter.log.output.enabled |
true | true: enable output log data;false: disable all log data output |
reporter.log.output.messageMaxBytes |
999900 | Maximum bytes sendable per message including encoding overhead. |
reporter.log.output.queuedMaxLogs |
1000 | Maximum backlog of logs reported before sent. |
reporter.log.output.queuedMaxSize |
1000000 | Maximum backlog of log bytes reported before sent. |
reporter.log.output.messageTimeout |
1000 | Logs are bundled into messages, up to messageMaxBytes. This timeout starts when the first unsent log is reported, which ensures that logs are not stuck in an incomplete message. |
Easeagent provides a header pass-through plugin.
Config format:
easeagent.progress.forwarded.headers.{key}={headerName}
- {key} indicates the unique key of the header configuration, used to identify the configuration modification
- {headerName} is the Header Name you need to pass through
Example:
easeagent.progress.forwarded.headers.canary.0=X-Mesh-CanaryIn the process of supporting easemesh traffic coloring, the request header X-Mesh-Canary needs to be deeply passed through.
(add header: X-Mesh-Canary=lv1) -> serviceA(X-Mesh-Canary=lv1) -> mesh(check X-Mesh-Canary) --> servcieB
|
|_____> servcieB-canary(X-Mesh-Canary=lv1)
plugin enabled config: Enabled
Easeagent will grab the header from the response of the process, and put the name and value of the header as a tag in the Span of Tracing.
Config format:
observability.tracings.tag.response.headers.{key}={value}
- {key} indicates the unique key of the header configuration, used to identify the configuration modification
- {headerName} is the Header Name you need to tag
Example:
observability.tracings.tag.response.headers.eg.0=X-EG-Circuit-BreakerIn the process of supporting sidecars (such as easemesh), the sidecars will hijack or color traffic according to the situation.
In order to facilitate observation and drawing, sidecars should add header information in the response header and record the tag in Tracing.
example:
easemesh adds the following header information: X-EG-Circuit-Breaker, X-EG-Retryer, X-EG-Rate-Limiter, X-EG-Time-Limiter
The tag will be added to the Tracing Span of the request client:
{"kind": "CLIENT", "tags": {"X-EG-Circuit-Breaker":"aaaa", "X-EG-Retryer":"bbbb", "X-EG-Rate-Limiter":"cccc", "X-EG-Time-Limiter":"dddd"}}Most capabilities of Easeagent, such as tracing and metric, are provided through plugins. The format of the plugin configuration is defined as follows.
plugin.[domain].[namespace].[function].[key] = [value]
Take the tracing switch of httpclient as an example.
plugin.observability.httpclient.tracing.enabled=true
domain : observability
namespace : httpclient
function : tracing
key : enabled
value : true
[domain] and [namespace] and [function] are defined by plugins, and further details can be found in the plugin development guide.
For plugin level configuration, EaseAgent defines a spacial namespace of global in which user can define default configuration for any function, like metric, and each namespace plugin of this function will uses the default configuration when it does not create configuration with its own namespace.
For example, Metric have a set of default plugin configuration as follows:
plugin.observability.global.metric.enabled=true
plugin.observability.global.metric.interval=30
plugin.observability.global.metric.topic=application-meter
plugin.observability.global.metric.appendType=kafka
All metric plugins will inherit default configuration, unless they have configured a configuration item with the same [key] and replaced the global namespace with its own namespace to override.
# this configuration item of rabbitmq indicate that rabbitmq's metirc data is printed to console instead of send to kafka server as configured by the default.
plugin.observability.rabbitmq.metric.appendType=console
But the switch configuration item using enabled as key cannot be overridden, for boolean type configuration is determined by a "logical AND" operation between the global and its own namespace configuration.
The following sections describe the metric and tracing configuration items, as well as the currently supported plugins and their corresponding namespaces
| Key | Default Value | Description |
|---|---|---|
plugin.observability.global.tracing.enabled |
true | Enable all tracing collection. false: Disable all tracing collection. |
plugin.observability.global.metric.enabled |
true | Enable all metrics collection. false: Disable all metrics collection. |
plugin.observability.global.metric.interval |
30 | Time interval between two outputs. Time Unit: second. |
plugin.observability.global.metric.topic |
application-meter | Send metric data to the specified kafka topic, only avaliable when appendType is kafka. |
plugin.observability.global.metric.url |
/metrics | Send metric data to the specified http URI, which will be appended to reporter.outputServer.bootstrapServer, to form a full url, only avaliable when appendType is http. |
plugin.observability.global.metric.appendType |
kafka | The value should be kafka, console or http. kafka: EaseAgent will output metric data to kafka server. console: EaseAgent will output metric data to console; http: output metric data to http server. |
Supported components and corresponding namespaces:
| Plugin/Components | Namespace | Description |
|---|---|---|
| httpservlet | httpServlet |
Http Request Metric |
| spring-gateway | springGateway |
Http Request Metric |
| jdbcConnection | jdbcConnection |
JDBC Connection Metric |
| jdbcStatement | jdbcStatement |
JDBC SQL Metric. When using SQL as a tag, the string length of SQL is often very long, which will consume network bandwidth and CPU to a great extent. Our solution is to use SQL's MD5 as an indicator, which is associated with the storage and front-end.Closed configuration: plugin.observability.jdbc.sql.compress.enabled=false |
| md5Dictionary | md5Dictionary |
SQL-MD5Dictionary. When EaseAgent is used with EaseMesh, tracing and metric data will be stored in Elasticsearch. In order to reduce the space occupied by SQL in Elasticsearch, EaseAgent uses md5 to reduce the length of SQL, and then periodically stores it in Kafka, and finally stores it in Elasticsearch. Only one copy of sql will be stored in Elasticsearch. |
| redis | redis |
Redis Metric |
| kafka | kafka |
Kafka Metric |
| rabbitmq | rabbitmq |
RabbitMQ Metric |
| jvmGc | jvmGc |
JVM GC Metric |
| JVM Memory | jvmMemory |
JVM Memory Metric |
| dubbo | dubbo |
dubbo Metric |
| motan | motan |
Motan Metric |
| sofarpc | sofarpc |
SOFARPC Metric |
Application log modules collecting application logs printed by the application.
Supported components/plugins and corresponding namespaces:
| Plugin/Components | Namespace | Description |
|---|---|---|
| logback | logback |
Support logback library |
| log4j2 | log4j2 |
Support log4j2 library |
| access | access |
Access log module |
The default configuration is as follows:
plugin.observability.global.log.enabled=true
plugin.observability.global.log.appendType=console
plugin.observability.global.log.topic=application-log
plugin.observability.global.log.url=/application-log
plugin.observability.global.log.level=INFO
plugin.observability.global.log.encoder=LogDataJsonEncoder
plugin.observability.global.log.encoder.timestamp=%d{UNIX_MILLIS}
plugin.observability.global.log.encoder.logLevel=%-5level
plugin.observability.global.log.encoder.threadId=%thread
plugin.observability.global.log.encoder.location=%logger{36}
plugin.observability.global.log.encoder.message=%msg%n
plugin.observability.access.log.encoder=AccessLogJsonEncoder
plugin.observability.logback.log.enabled=false
plugin.observability.log4j2.log.enabled=false
The logback and log4j2 modules are disabled by default, and they can be enabled in the user configuration file to enable collecting logs printed by the application.
The LogDataJsonEncoder supports log4j2 style pattern configuration for each field.
To send logs data to Opentelemetry compatible backend, the corresponding Encoder need to be developed.
Redirection feature combined with EaseMesh to direct traffic to shadow services to simulate real traffic for the whole site performance test in the production environment in an effective and safe way.
For more detail, please reference EaseMesh documents.
The default configuration has only one item:
plugin.integrability.global.redirect.enabled=true
Supported components/plugins and corresponding namespaces:
| Plugin/Components | Namespace | Description |
|---|---|---|
| jdbc | jdbc |
Database Redirection |
| redis | redis |
Redis Redirection |
| kafka | kafka |
Kafka Redirection |
| rabbitmq | rabbitmq |
RabbitMQ Redirection |
| elasticsearch | elasticsearch |
Elasticsearch Redirection |
Easeagent provides a header pass-through plugin.
plugin.integrability.global.forwarded.enabled=true
To support easemesh, we have added a new plugin called "servicename".
It will get the service name in advance, and then put the service name in the HTTP request header.
header name config:
plugin.integrability.serviceName.addServiceNameHead.propagate.head=X-Mesh-RPC-ServiceThe current way to obtain ServiceName only supports service discovery using Spring Cloud.
After the EaseAgent enabled the http port, the http api can be used to modify the configuration of the plugin.
-
The plugin configuration items can be modified directly from the configuration information mapping:
GET /plugins/domains/{domain}/namespaces/{namespace}/{id}/properties/{property}/{value}/{version} -
API supports passing json one-time "modification/addition" content instead of setting them one by one. For example:
POST /plugins/domains/{domain}/namespaces/{namespace}/{id}/properties { "version": "1", "property1": "value1", "property2": "value2" }
the {version} can be any information
EaseAgent use Log4j2 for all internal logging, the default log level is INFO, and the logs will be outputted to the Console. User can modify the log level and appender in the easeagent-log4j2.xml file.
After modification, User can run the application with EaseAgent.
$ export EASE_AGENT_PATH=[Replace with agent path]
$ java "-javaagent:${EASE_AGENT_PATH}/easeagent.jar -Deaseagent.log.conf=${EASE_AGENT_PATH}/easeagent-log4j2.xml" -jar user-app.jar
or
$ export EASE_AGENT_PATH=[Replace with agent path]
$ export EASEAGENT_LOG_CONF=/your/log4j2/config/filepath
$ java "-javaagent:${EASE_AGENT_PATH}/easeagent.jar -jar user-app.jar
Easeagent automatically adds TraceId and SpanId to the MDC (Mapped Diagnostic Context) when creating a new Span. You can configure your slf4j or logback files by adding parameters to display these IDs, or retrieve them directly from the MDC in your code.
Here is an example configuration file that displays traceId and spaceId in the log output::
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>!-%d{HH:mm:ss.SSS} [%thread] %-5level %logger{16} - [%X{traceId}/%X{spanId}] %msg%n</pattern>
</encoder>
</appender>Here is an example of how to retrieve traceId and spanId in code.
String traceId = org.slf4j.MDC.get("traceId");
String spanId = org.slf4j.MDC.get("spanId");When Internal HTTP Server is enabled, User can use Prometheus to collect metrics information.
- Adding the following configuration in
prometheus.yml
- job_name: 'user-app'
static_configs:
- targets: ['localhost:9900']
metrics_path: "/prometheus/metrics"
EaseAgent supply the health check、readiness check endpoint.
Health CheckEndpoint
[GET] http://[ip]:[easeagent.server.port]/health
The response status will be 200(OK)
Readiness CheckEndpoint After Spring sendingApplicationReadyEvent, EaseAgent will change readiness status totrue
[GET] http://[ip]:[easeagent.server.port]/health/readiness
The response status will be 200(OK)
EaseAgent supply the agent info http api.
[GET] http://[ip]:[easeagent.server.port]/agent-info
The response status will be 200(OK) Response Body:
{
"type": "EaseAgent",
"version": "x.x.x"
}EaseAgent use brave to collect tracing logs.The data format stored in Kafka is Zipkin Data Model. User can send tracing logs to Zipkin server.
| Component Type | Component | Reference |
|---|---|---|
| HTTP Client | RestTemplate、 WebClient、 FeignClient |
brave-instrumentation-http |
| HTTP Server | Servlet、Filter |
brave-instrumentation-http |
| DataBase | JDBC |
Brave |
| Cache | Jedis、Lettuce |
Brave |
| Message | RabbitMQ、Kafka |
brave-instrumentation-messaging 、Brave Kafka instrumentation |
| Logging | Log4j2、Logback |
brave-context-log4j2 、brave-context-slf4j |
| RPC | AlibabaDubbo、ApacheDubbo、Motan,SOFARPC |
brave-instrumentation-dubbo 、brave-instrumentation-dubbo-rpc、brave-instrumentation-rpc |
| Component Type | Component | Description |
|---|---|---|
| RPC | Motan |
The motan plugin adds a dynamic switch to collect interface parameters and return values for troubleshooting purposes. The display format is json (Notes: The library used is Jackson), which is disabled by default. You can enable the parameter collection switch with this configuration: plugin.observability.motan.tracing.args.collect.enabled=ture, and the return values collection switch with this configuration: plugin.observability.motan.tracing.result.collect.enabled=ture. |
| RPC | Dubbo |
The motan plugin adds a dynamic switch to collect interface parameters and return values for troubleshooting purposes. The display format is json (Notes: The library used is Jackson), which is disabled by default. You can enable the parameter collection switch with this configuration: plugin.observability.dubbo.tracing.args.collect.enabled=ture, and the return values collection switch with this configuration: plugin.observability.dubbo.tracing.result.collect.enabled=ture. |
| RPC | SOFARPC |
The motan plugin adds a dynamic switch to collect interface parameters and return values for troubleshooting purposes. The display format is json (Notes: The library used is Jackson), which is disabled by default. You can enable the parameter collection switch with this configuration: plugin.observability.sofarpc.tracing.args.collect.enabled=ture, and the return values collection switch with this configuration: plugin.observability.sofarpc.tracing.result.collect.enabled=ture. |
| Tag | Description |
|---|---|
| sql | Sql text in user application |
| local-component | Default value = 'database' |
| url | Connection information. Example: jdbc:mysql://localhost:3306/db_demo |
| error | SQLException information |
| Tag | Description |
|---|---|
| redis.method | Redis command. Example: MGET、GET |
| Tag | Description |
|---|---|
| rabbit.exchange | RabbitMQ exchange |
| rabbit.routing_key | RabbitMQ routingKey |
| rabbit.queue | RabbitMQ routingKey |
| Tag | Description |
|---|---|
| kafka.key | Kafka consumer record Key |
| kafka.topic | Kafka topic |
| kafka.broker | Kafka url |
| Tag | Description |
|---|---|
| dubbo.args | dubbo interface arguments |
| dubbo.result | dubbo interface return value |
| dubbo.group | dubbo client group name |
| dubbo.service | dubbo service interface full name |
| dubbo.method | dubbo service method signature |
| dubbo.service.version | dubbo service interface version |
| dubbo.client.application | dubbo client application name |
| dubbo.server.application | dubbo server application name |
| Tag | Description |
|---|---|
| motan.args | motan interface arguments |
| motan.result | motan interface return value |
| motan.service | motan service interface full name |
| motan.method | motan service method signature |
| motan.service.version | motan service interface version |
| motan.application | motan client application name |
| motan.module | motan server module name |
| motan.group | motan client group name |
| Tag | Description |
|---|---|
| sofarpc.args | SOFARPC interface arguments |
| sofarpc.result | SOFARPC interface return value |
| sofarpc.service | SOFARPC service interface full name |
| sofarpc.method | SOFARPC service method signature |
| sofarpc.service.uniqueId | SOFARPC service interface uniqueId |
| sofarpc.client.application | SOFARPC client application name |
| sofarpc.server.application | SOFARPC server module name |
EaseAgent use io.dropwizard.metrics to collect metric information.
Prometheus Metric Schedule: Prometheus Metric
Prometheus Exports Rules: Prometheus Exports
EaseAgent output metric data to kafka. The data stored in kafka is in JSON format.
For Example: EaseAgent collect metric of HTTP Request. The collected metric data are as follows:
{
"m15err" : 0,
"m5err" : 0,
"cnt" : 1,
"url" : "GET \/",
"m5" : 0.050990000000000001,
"max" : 823,
"mean" : 823,
"p98" : 823,
"errcnt" : 0,
"host_name" : "akwei",
"min" : 823,
"category" : "application",
"system" : "none",
"type" : "http-request",
"mean_rate" : 0,
"p99" : 823,
"p95" : 823,
"m15" : 0.12681999999999999,
"timestamp" : 1621567320892,
"service" : "unknown-service",
"m1" : 0.00022000000000000001,
"m5errpct" : 0,
"p25" : 823,
"p75" : 823,
"p50" : 823,
"host_ipv4" : "192.168.2.5",
"m1errpct" : 0,
"m15errpct" : 0,
"m1err" : 0,
"p999" : 823
}For different kind of metrics, we have different schemas:
HTTP Request schema describes key metrics of service APIs, which include:
- Total execution count (cnt, errcnt)
- Throughput (m1, m5, m15)
- Error throughput (m1err, m5err, m15err)
- Error throughput percentage (m1errpct, m5errpct, m15errpct)
- Latency (p25, p50, p75, p95, p98, p99)
- Execution duration (min, mean, max)
| Field | Type | Description |
|---|---|---|
| url | string | the URL of the request |
| cnt | integer | The total count of the request executed |
| errcnt | integer | The total error count of the request executed |
| m1 | double | The HTTP request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The HTTP request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The HTTP request executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The HTTP error request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The HTTP error request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The HTTP error request executions per second (exponentially-weighted moving average) in last 15 minute |
| m1errpct | double | error percentage in last 1 minute |
| m5errpct | double | error percentage in last 5 minute |
| m15errpct | double | error percentage in last 15 minute |
| min | double | The http-request minimal execution duration in milliseconds. |
| max | double | The http-request maximal execution duration in milliseconds. |
| mean | double | The http-request mean execution duration in milliseconds. |
| p25 | double | TP25: The http-request execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The http-request execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The http-request execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The http-request execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The http-request execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The http-request execution duration in milliseconds for 99% user. |
JDBC Statement schema describes key metrics of JDBC SQL Statement, which include:
- Execution count (cnt, errcnt)
- Throughput (m1, m5, m15)
- Error throughput (m1err, m5err, m15err)
- Latency (p25, p50, p75, p95, p98, p99, p999)
- Execution duration (min, mean, max)
| Field | Type | Description |
|---|---|---|
| signature | string | Executed JDBC method signature. |
| cnt | integer | The total count of JDBC method executed |
| errcnt | integer | The total error count of JDBC method executed |
| m1 | double | The JDBC method executions per second (exponentially-weighted moving average) in last 1 minute. |
| m5 | double | The JDBC method executions per second (exponentially-weighted moving average) in last 5 minutes. |
| m15 | double | The JDBC method executions per second (exponentially-weighted moving average) in last 15 minutes. |
| m1err | double | The JDBC method error executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The JDBC method error executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The JDBC method error executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The JDBC method minimal execution duration in milliseconds. |
| max | double | The JDBC method maximal execution duration in milliseconds. |
| mean | double | The JDBC method mean execution duration in milliseconds. |
| p25 | double | TP25: The JDBC method execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The JDBC method execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The JDBC method execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The JDBC method execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The JDBC method execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The JDBC method execution duration in milliseconds for 99% user. |
| p999 | double | TP99.9: The JDBC method execution duration in milliseconds for 99.9% user. |
JDBC Connection schema describes key metrics of Getting Connection, which include:
- Execution count (cnt, errcnt)
- Throughput (m1, m5, m15)
- Error throughput (m1err, m5err, m15err)
- Latency (p25, p50, p75, p95, p98, p99, p999)
- Execution duration (min, mean, max)
| Field | Type | Description |
|---|---|---|
| url | string | The url of database connections |
| cnt | integer | The total number of database connections |
| errcnt | integer | The total error number of database connections |
| m1 | double | The JDBC connection establishment per second (exponentially-weighted moving average) in last 1 minute. |
| m5 | double | The JDBC connection establishment per second (exponentially-weighted moving average) in last 5 minutes. |
| m15 | double | The JDBC connection establishment per second (exponentially-weighted moving average) in last 15 minutes. |
| m1err | double | The JDBC connection error executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The JDBC connection error executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The JDBC connection error executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The JDBC connection minimal establishment duration in milliseconds. |
| max | double | The JDBC connection maximal establishment duration in milliseconds. |
| mean | double | The JDBC connection mean establishment duration in milliseconds. |
| p25 | double | TP25: The JDBC connection establishment duration in milliseconds for 25% user. |
| p50 | double | TP50: The JDBC connection establishment duration in milliseconds for 50% user. |
| p75 | double | TP75: The JDBC connection establishment duration in milliseconds for 75% user. |
| p95 | double | TP95: The JDBC connection establishment duration in milliseconds for 95% user. |
| p98 | double | TP98: The JDBC connection establishment duration in milliseconds for 98% user. |
| p99 | double | TP99: The JDBC connection establishment duration in milliseconds for 99% user. |
| p999 | double | TP99.9: The JDBC connection establishment duration in milliseconds for 99.9% user. |
JVM Memory schema describes key metrics of Java memory usage, which include:
- bytes-init
- bytes-used
- bytes-committed
- bytes-max
| Field | Type | Description |
|---|---|---|
| resource | String | memory pool name |
| bytes-init | integer | The value represents the initial amount of memory in bytes unit that the JVM requests from the operating system for memory management during startup. The JVM may request additional memory from the operating system and may also release memory to the system over time. The value of init may be undefined (value -1). |
| bytes-used | integer | The value represents the amount of memory currently used in bytes unit. |
| bytes-committed | integer | The value represents the amount of memory in bytes unit that is guaranteed to be available for use by the JVM. The amount of committed memory may change over time (increase or decrease). The JVM may release memory to the system and committed could be less than init. Value committed will always be greater than or equal to used. |
| bytes-max | integer | The value represents the maximum amount of memory in bytes unit that can be used for memory management. Its value may be undefined (value -1). The maximum amount of memory may change over time if defined. The amount of used and committed memory will always be less than or equal to max if max is defined. A memory allocation may fail if it attempts to increase the used memory such that used > committed even if used <= max would still be true (for example, when the system is low on virtual memory). |
JVM GC schema describes key metrics of JVM garbage collection, which include:
- total_collection_time
- times
- times_rate
| Field | Type | Description |
|---|---|---|
| resource | string | gc name |
| total_collection_time | integer | The value represents the total time for garbage collection operation in millisecond unit. |
| times | integer | The value represents the total garbage collection times. |
| times_rate | integer | The number of gc times per second. |
Kafka Client schema describes key metrics of Kafka client invoking, which include:
- Producer
- Throughput (prodrm1, prodrm5, prodrm15)
- Error throughput (prodrm1err, prodrm5err, prodrm15err)
- Execution duration (prodrmin, prodrmean, prodrmax)
- Latency (prodrp25, prodrp50, prodrp75, prodrp95, prodrp98, prodrp99, prodrp999)
- Consumer
- Throughput (consrm1, consrm5, consrm15)
- Error throughput (consrm1err, consrm5err, consrm15err)
- Execution duration (consrmin, consrmean, consrmax)
- Latency (consrp25, consrp50, consrp75, consrp95, consrp98, consrp99, consrp999)
| Field | Type | Description |
|---|---|---|
| resource | string | topic name |
| prodrm1 | double | The executions per second (exponentially-weighted moving average) in last 1 minute (producer) |
| prodrm5 | double | The executions per second (exponentially-weighted moving average) in last 5 minute (producer) |
| prodrm15 | double | The executions per second (exponentially-weighted moving average) in last 15 minute (producer) |
| consrm1 | double | The executions per second (exponentially-weighted moving average) in last 1 minute (consumer) |
| consrm5 | double | The executions per second (exponentially-weighted moving average) in last 5 minute (consumer) |
| consrm15 | double | The executions per second (exponentially-weighted moving average) in last 15 minute (consumer) |
| prodrm1err | double | The error executions per second (exponentially-weighted moving average) in last 1 minute (producer) |
| prodrm5err | double | The executions per second (exponentially-weighted moving average) in last 5 minute (producer) |
| prodrm5err | double | The error executions per second (exponentially-weighted moving average) in last 15 minute (producer) |
| consrm1err | double | The error executions per second (exponentially-weighted moving average) in last 1 minute (consumer) |
| consrm5err | double | The error executions per second (exponentially-weighted moving average) in last 5 minute (consumer) |
| consrm5err | double | The error executions per second (exponentially-weighted moving average) in last 15 minute (consumer) |
| prodrmin | double | The minimal execution duration in milliseconds. |
| prodrmax | double | The maximal execution duration in milliseconds. |
| prodrmean | double | The mean execution duration in milliseconds. |
| prodrp25 | double | TP25: The execution duration in milliseconds for 25% user. |
| prodrp50 | double | TP50: The execution duration in milliseconds for 50% user. |
| prodrp75 | double | TP75: The execution duration in milliseconds for 75% user. |
| prodrp95 | double | TP95: The execution duration in milliseconds for 95% user. |
| prodrp98 | double | TP98: The execution duration in milliseconds for 98% user. |
| prodrp99 | double | TP99: The execution duration in milliseconds for 99% user. |
| prodrp999 | double | TP99.9: The execution duration in milliseconds for 99.9% user. |
| consrmin | double | The minimal execution duration in milliseconds. |
| consrmax | double | The maximal execution duration in milliseconds. |
| consrmean | double | The mean execution duration in milliseconds. |
| consrp25 | double | TP25: The execution duration in milliseconds for 25% user. |
| consrp50 | double | TP50: The execution duration in milliseconds for 50% user. |
| consrp75 | double | TP75: The execution duration in milliseconds for 75% user. |
| consrp95 | double | TP95: The execution duration in milliseconds for 95% user. |
| consrp98 | double | TP98: The execution duration in milliseconds for 98% user. |
| consrp99 | double | TP99: The execution duration in milliseconds for 99% user. |
| consrp999 | double | TP99.9: The execution duration in milliseconds for 99.9% user. |
RabbitMQ Producer schema describes key metrics of RabbitMQ client publishing message, which include:
- Throughput (prodrm1, prodrm5, prodrm15)
- Error throughput (prodrm1err, prodrm5err, prodrm15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99)
| Field | Type | Description |
|---|---|---|
| resource | string | rabbitmq exchange or routingkey |
| prodrm1 | double | The executions of producer per second (exponentially-weighted moving average) in last 1 minute |
| prodrm5 | double | The executions of producer per second (exponentially-weighted moving average) in last 5 minute |
| prodrm15 | double | The executionsof producer per second (exponentially-weighted moving average) in last 15 minute |
| prodrm1err | double | The error executions per second (exponentially-weighted moving average) in last 1 minute (producer) |
| prodrm5err | double | The executions per second (exponentially-weighted moving average) in last 5 minute (producer) |
| prodrm5err | double | The error executions per second (exponentially-weighted moving average) in last 15 minute (producer) |
| min | double | The producer minimal execution duration in milliseconds. |
| max | double | The producer maximal execution duration in milliseconds. |
| mean | double | The producer mean execution duration in milliseconds. |
| p25 | double | TP25: The producer execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The producer execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The producer execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The producer execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The producer execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The producer execution duration in milliseconds for 99% user. |
| p999 | double | TP99.9: The execution duration in milliseconds for 99.9% user. |
RabbitMQ Consumer schema describes key metrics of RabbitMQ client consuming message, which include:
- Throughput (queue_m1_rate, queue_m5_rate, queue_m15_rate)
- Error throughput (queue_m1_error_rate, queue_m5_error_rate, queue_m15_error_rate)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99)
| Field | Type | Description |
|---|---|---|
| resource | string | rabbitmq routingKey |
| queue_m1_rate | double | The executions of queue per second (exponentially-weighted moving average) in last 1 minute |
| queue_m5_rate | double | The executions of queue per second (exponentially-weighted moving average) in last 5 minute |
| queue_m15_rate | double | The executionsof queue per second (exponentially-weighted moving average) in last 15 minute |
| queue_m1_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 1 minute (queue) |
| queue_m5_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 5 minute (queue) |
| queue_m15_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 15 minute (queue) |
| min | double | The consumer minimal execution duration in milliseconds. |
| max | double | The consumer maximal execution duration in milliseconds. |
| mean | double | The consumer mean execution duration in milliseconds. |
| p25 | double | TP25: The consumer execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The consumer execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The consumer execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The consumer execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The consumer execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The consumer execution duration in milliseconds for 99% user. |
| p999 | double | TP99.9: The execution duration in milliseconds for 99.9% user. |
Message Listener schema describes key metrics of Spring AMQP RabbitMQ Message Queue, which include:
- Throughput (queue_m1_rate, queue_m5_rate, queue_m15_rate)
- Error throughput (queue_m1_error_rate, queue_m5_error_rate, queue_m15_error_rate)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99)
| Field | Type | Description |
|---|---|---|
| resource | string | rabbitmq queue |
| queue_m1_rate | double | The executions of queue per second (exponentially-weighted moving average) in last 1 minute |
| queue_m5_rate | double | The executions of queue per second (exponentially-weighted moving average) in last 5 minute |
| queue_m15_rate | double | The executionsof queue per second (exponentially-weighted moving average) in last 15 minute |
| queue_m1_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 1 minute (queue) |
| queue_m5_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 5 minute (queue) |
| queue_m15_error_rate | double | The error executions per second (exponentially-weighted moving average) in last 15 minute (queue) |
| min | double | The AMQP Message Listener minimal execution duration in milliseconds. |
| max | double | The AMQP Message Listener maximal execution duration in milliseconds. |
| mean | double | The AMQP Message Listener mean execution duration in milliseconds. |
| p25 | double | TP25: The AMQP Message Listener execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The AMQP Message Listener execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The AMQP Message Listener execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The AMQP Message Listener execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The AMQP Message Listener execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The AMQP Message Listener execution duration in milliseconds for 99% user. |
| p999 | double | TP99.9: The execution duration in milliseconds for 99.9% user. |
Elasticsearch schema describes key metrics of Elasticsearch client invoking, which include:
- Total execution count (cnt, errcnt, m1cnt, m5cnt, m15cnt)
- Throughput (m1, m5, m15, mean_rate)
- Error throughput (m1err, m5err, m15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99)
| Field | Type | Description |
|---|---|---|
| index | string | The Elasticsearch index name |
| cnt | integer | The total count of the request executed |
| errcnt | integer | The total error count of the request executed |
| m1cnt | integer | The total count of the request executed in last 1 minute |
| m5cnt | integer | The total count of the request executed in last 5 minute |
| m15cnt | integer | The total count of the request executed in last 15 minute |
| m1 | double | The Elasticsearch request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The Elasticsearch request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The Elasticsearch request executions per second (exponentially-weighted moving average) in last 15 minute. |
| mean_rate | double | The Elasticsearch request executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The Elasticsearch error request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The Elasticsearch error request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The Elasticsearch error request executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The Elasticsearch minimal execution duration in milliseconds. |
| max | double | The Elasticsearch maximal execution duration in milliseconds. |
| mean | double | The Elasticsearch mean execution duration in milliseconds. |
| p25 | double | TP25: The Elasticsearch execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The Elasticsearch execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The Elasticsearch execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The Elasticsearch execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The Elasticsearch execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The Elasticsearch execution duration in milliseconds for 99% user. |
MongoDB schema describes key metrics of MongoDB client invoking, which include:
- Total execution count (cnt, errcnt, m1cnt, m5cnt, m15cnt)
- Throughput (m1, m5, m15, mean_rate)
- Error throughput (m1err, m5err, m15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99)
| Field | Type | Description |
|---|---|---|
| operation | string | The MongoDB request command name |
| cnt | integer | The total count of the request executed |
| errcnt | integer | The total error count of the request executed |
| m1cnt | integer | The total count of the request executed in last 1 minute |
| m5cnt | integer | The total count of the request executed in last 5 minute |
| m15cnt | integer | The total count of the request executed in last 15 minute |
| m1 | double | The MongoDB request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The MongoDB request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The MongoDB request executions per second (exponentially-weighted moving average) in last 15 minute. |
| mean_rate | double | The MongoDB request executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The MongoDB error request executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The MongoDB error request executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The MongoDB error request executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The MongoDB minimal execution duration in milliseconds. |
| max | double | The MongoDB maximal execution duration in milliseconds. |
| mean | double | The MongoDB mean execution duration in milliseconds. |
| p25 | double | TP25: The MongoDB execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The MongoDB execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The MongoDB execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The MongoDB execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The MongoDB execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The MongoDB execution duration in milliseconds for 99% user. |
Dubbo schema describes key metrics of Dubbo client invoking, which include:
- Total execution count (cnt, errcnt, m1cnt, m5cnt, m15cnt)
- Throughput (m1, m5, m15, mean_rate)
- Error throughput (m1err, m5err, m15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99, p999)
| Field | Type | Description |
|---|---|---|
| interface | string | Dubbo full method signature. |
| cnt | integer | The total count of the Dubbo method executed |
| errcnt | integer | The total error count of the Dubbo method executed |
| m1cnt | integer | The total count of the Dubbo method executed in last 1 minute |
| m5cnt | integer | The total count of the Dubbo method executed in last 5 minute |
| m15cnt | integer | The total count of the Dubbo method executed in last 15 minute |
| m1 | double | The Dubbo method executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The Dubbo method executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The Dubbo method executions per second (exponentially-weighted moving average) in last 15 minute. |
| mean_rate | double | The Dubbo method executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The Dubbo method error executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The Dubbo method error executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The Dubbo method error executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The Dubbo method minimal execution duration in milliseconds. |
| max | double | The Dubbo method maximal execution duration in milliseconds. |
| mean | double | The Dubbo method mean execution duration in milliseconds. |
| p25 | double | TP25: The Dubbo method execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The Dubbo method execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The Dubbo method execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The Dubbo method execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The Dubbo method execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The Dubbo method execution duration in milliseconds for 99% user. |
| p999 | double | TP999: The Dubbo method execution duration in milliseconds for 99.9% user. |
Motan schema describes key metrics of Motan client invoking, which include:
- Total execution count (cnt, errcnt, m1cnt, m5cnt, m15cnt)
- Throughput (m1, m5, m15, mean_rate)
- Error throughput (m1err, m5err, m15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99, p999)
| Field | Type | Description |
|---|---|---|
| interface | string | Motan full method signature. |
| cnt | integer | The total count of the Motan method executed |
| errcnt | integer | The total error count of the Motan method executed |
| m1cnt | integer | The total count of the Motan method executed in last 1 minute |
| m5cnt | integer | The total count of the Motan method executed in last 5 minute |
| m15cnt | integer | The total count of the Motan method executed in last 15 minute |
| m1 | double | The Motan method executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The Motan method executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The Motan method executions per second (exponentially-weighted moving average) in last 15 minute. |
| mean_rate | double | The Motan method executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The Motan method error executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The Motan method error executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The Motan method error executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The Motan method minimal execution duration in milliseconds. |
| max | double | The Motan method maximal execution duration in milliseconds. |
| mean | double | The Motan method mean execution duration in milliseconds. |
| p25 | double | TP25: The Motan method execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The Motan method execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The Motan method execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The Motan method execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The Motan method execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The Motan method execution duration in milliseconds for 99% user. |
| p999 | double | TP999: The Motan method execution duration in milliseconds for 99.9% user. |
SOFARPC schema describes key metrics of SOFARPC client invoking, which include:
- Total execution count (cnt, errcnt, m1cnt, m5cnt, m15cnt)
- Throughput (m1, m5, m15, mean_rate)
- Error throughput (m1err, m5err, m15err)
- Execution duration (min, mean, max)
- Latency (p25, p50, p75, p95, p98, p99, p999)
| Field | Type | Description |
|---|---|---|
| interface | string | SOFARPC method signature. |
| cnt | integer | The total count of the SOFARPC method executed |
| errcnt | integer | The total error count of the SOFARPC method executed |
| m1cnt | integer | The total count of the SOFARPC method executed in last 1 minute |
| m5cnt | integer | The total count of the SOFARPC method executed in last 5 minute |
| m15cnt | integer | The total count of the SOFARPC method executed in last 15 minute |
| m1 | double | The SOFARPC method executions per second (exponentially-weighted moving average) in last 1 minute |
| m5 | double | The SOFARPC method executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15 | double | The SOFARPC method executions per second (exponentially-weighted moving average) in last 15 minute. |
| mean_rate | double | The SOFARPC method executions per second (exponentially-weighted moving average) in last 15 minute. |
| m1err | double | The SOFARPC method error executions per second (exponentially-weighted moving average) in last 1 minute |
| m5err | double | The SOFARPC method error executions per second (exponentially-weighted moving average) in last 5 minute. |
| m15err | double | The SOFARPC method error executions per second (exponentially-weighted moving average) in last 15 minute |
| min | double | The SOFARPC method minimal execution duration in milliseconds. |
| max | double | The SOFARPC method maximal execution duration in milliseconds. |
| mean | double | The SOFARPC method mean execution duration in milliseconds. |
| p25 | double | TP25: The SOFARPC method execution duration in milliseconds for 25% user. |
| p50 | double | TP50: The SOFARPC method execution duration in milliseconds for 50% user. |
| p75 | double | TP75: The SOFARPC method execution duration in milliseconds for 75% user. |
| p95 | double | TP95: The SOFARPC method execution duration in milliseconds for 95% user. |
| p98 | double | TP98: The SOFARPC method execution duration in milliseconds for 98% user. |
| p99 | double | TP99: The SOFARPC method execution duration in milliseconds for 99% user. |
| p999 | double | TP999: The SOFARPC method execution duration in milliseconds for 99.9% user. |