TRACE_TOO_LARGE getting error in grafana tempo 2.3.0

[raagulrakesh]

27T09:39:36.593764703Z caller=server.go:1184 method=/tempopb.Pusher/PushBytesV2 duration=66.116µs msg=gRPC err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 861982 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"
level=error ts=2024-05-27T09:39:36.594330685Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 861982 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"
level=warn ts=2024-05-27T09:39:37.003237927Z caller=server.go:1184 method=/tempopb.Pusher/PushBytesV2 duration=75.081µs msg=gRPC err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 670483 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"
level=error ts=2024-05-27T09:39:37.003433076Z caller=rate_limited_logger.go:27 msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 670483 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"
level=warn ts=2024-05-27T09:39:37.66943521Z caller=server.go:1184 method=/tempopb.Pusher/PushBytesV2 duration=76.276µs msg=gRPC err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 393874 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"

[raagulrakesh]

CONFIGMAP

{
"overrides.yaml": "overrides:
defaults:
global:
max_bytes_per_trace: 50000000
",
"tempo.yaml": "multitenancy_enabled: false
usage_report:
reporting_enabled: true
compactor:
compaction:
block_retention: 720h
distributor:
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_binary:
endpoint: 0.0.0.0:6832
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
opencensus: null
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
ingester:
{}
server:
http_listen_port: 3100
storage:
trace:
backend: local
local:
path: /var/tempo/traces
wal:
path: /var/tempo/wal
querier:
{}
query_frontend:
{}
overrides:
per_tenant_override_config: /conf/overrides.yaml
"
}

[mapno]

Ingestion limits are configured via overrides. See https://grafana.com/docs/tempo/latest/configuration/#ingestion-limits. You're looking for max_bytes_per_trace in particular.

[raagulrakesh]

Hi

I didn't see max_bytes_per_trace in Ingestion limits. I am able to see in override global config. I have already added this global config.

"overrides.yaml": "overrides:
	  defaults:
	     global:
	       max_bytes_per_trace: 50000000

still seeing below error in tempo logs

27T09:39:36.593764703Z caller=server.go:1184 method=/tempopb.Pusher/PushBytesV2 duration=66.116µs msg=gRPC err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 861982 bytes to trace 0ecd3f27b25aaf415caa4c6009efaca4 for tenant single-tenant"

[mapno]

You can keep incrementing that parameter.

Although 50mb is already on the high end, you'll likely start experiencing issues rendering traces much bigger than that. I recommend you try to keep traces below 60-70mb in size.

[raagulrakesh]

I have increased max_bytes_per_trace to 50MB. but still showing as TRACE_TOO_LARGE: max size of trace (5000000). it is 5000000=5MB

[mapno]

Hmm, it looks like it's using the default value. For some reason your overrides config is not being applied. Can you go to /status/config and see what config is being picked up by Tempo?

[raagulrakesh]

Hi ,

Sorry for late reply
I have curl the http://localhost:3100/status/config and I get the below data

GET /status/config

target: all
http_api_prefix: ""
server:
http_listen_network: tcp
http_listen_address: ""
http_listen_port: 3100
http_listen_conn_limit: 0
grpc_listen_network: tcp
grpc_listen_address: ""
grpc_listen_port: 9095
grpc_listen_conn_limit: 0
tls_cipher_suites: ""
tls_min_version: ""
http_tls_config:
cert: ""
key: null
client_ca: ""
cert_file: ""
key_file: ""
client_auth_type: ""
client_ca_file: ""
grpc_tls_config:
cert: ""
key: null
client_ca: ""
cert_file: ""
key_file: ""
client_auth_type: ""
client_ca_file: ""
register_instrumentation: true
graceful_shutdown_timeout: 30s
http_server_read_timeout: 30s
http_server_write_timeout: 30s
http_server_idle_timeout: 2m0s
grpc_server_max_recv_msg_size: 16777216
grpc_server_max_send_msg_size: 16777216
grpc_server_max_concurrent_streams: 100
grpc_server_max_connection_idle: 2562047h47m16.854775807s
grpc_server_max_connection_age: 2562047h47m16.854775807s
grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
grpc_server_keepalive_time: 2h0m0s
grpc_server_keepalive_timeout: 20s
grpc_server_min_time_between_pings: 10s
grpc_server_ping_without_stream_allowed: true
log_format: logfmt
log_level: info
log_source_ips_enabled: false
log_source_ips_header: ""
log_source_ips_regex: ""
log_request_headers: false
log_request_at_info_level_enabled: false
log_request_exclude_headers_list: ""
http_path_prefix: ""
internal_server:
http_listen_network: tcp
http_listen_address: localhost
http_listen_port: 3101
http_listen_conn_limit: 0
grpc_listen_network: ""
grpc_listen_address: ""
grpc_listen_port: 0
grpc_listen_conn_limit: 0
tls_cipher_suites: ""
tls_min_version: ""
http_tls_config:
cert: ""
key: null
client_ca: ""
cert_file: ""
key_file: ""
client_auth_type: ""
client_ca_file: ""
grpc_tls_config:
cert: ""
key: null
client_ca: ""
cert_file: ""
key_file: ""
client_auth_type: ""
client_ca_file: ""
register_instrumentation: false
graceful_shutdown_timeout: 30s
http_server_read_timeout: 30s
http_server_write_timeout: 30s
http_server_idle_timeout: 2m0s
grpc_server_max_recv_msg_size: 0
grpc_server_max_send_msg_size: 0
grpc_server_max_concurrent_streams: 0
grpc_server_max_connection_idle: 0s
grpc_server_max_connection_age: 0s
grpc_server_max_connection_age_grace: 0s
grpc_server_keepalive_time: 0s
grpc_server_keepalive_timeout: 0s
grpc_server_min_time_between_pings: 0s
grpc_server_ping_without_stream_allowed: false
log_format: logfmt
log_level: info
log_source_ips_enabled: false
log_source_ips_header: ""
log_source_ips_regex: ""
log_request_headers: false
log_request_at_info_level_enabled: false
log_request_exclude_headers_list: ""
http_path_prefix: ""
enable: false
distributor:
ring:
kvstore:
store: memberlist
prefix: collectors/
consul:
host: localhost:8500
acl_token: ""
http_client_timeout: 20s
consistent_reads: false
watch_rate_limit: 1
watch_burst_size: 1
cas_retry_delay: 1s
etcd:
endpoints: []
dial_timeout: 10s
max_retries: 10
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
username: ""
password: ""
multi:
primary: ""
secondary: ""
mirror_enabled: false
mirror_timeout: 2s
heartbeat_period: 5s
heartbeat_timeout: 5m0s
instance_id: tempo-0
instance_interface_names:
- eth0
- en0
instance_port: 0
instance_addr: ""
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_binary:
endpoint: 0.0.0.0:6832
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
opencensus: null
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
override_ring_key: distributor
forwarders: []
extend_writes: true
retry_after_on_resource_exhausted: 0s
ingester_client:
pool_config:
checkinterval: 15s
healthcheckenabled: true
healthchecktimeout: 1s
maxconcurrenthealthchecks: 0
remote_timeout: 5s
grpc_client_config:
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
grpc_compression: snappy
rate_limit: 0
rate_limit_burst: 0
backoff_on_ratelimits: false
backoff_config:
min_period: 100ms
max_period: 10s
max_retries: 10
initial_stream_window_size: 63KiB1023B
initial_connection_window_size: 63KiB1023B
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
connect_timeout: 5s
connect_backoff_base_delay: 1s
connect_backoff_max_delay: 5s
metrics_generator_client:
pool_config:
checkinterval: 15s
healthcheckenabled: true
healthchecktimeout: 1s
maxconcurrenthealthchecks: 0
remote_timeout: 5s
grpc_client_config:
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
grpc_compression: snappy
rate_limit: 0
rate_limit_burst: 0
backoff_on_ratelimits: false
backoff_config:
min_period: 100ms
max_period: 10s
max_retries: 10
initial_stream_window_size: 63KiB1023B
initial_connection_window_size: 63KiB1023B
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
connect_timeout: 5s
connect_backoff_base_delay: 1s
connect_backoff_max_delay: 5s
querier:
search:
query_timeout: 30s
prefer_self: 10
external_hedge_requests_at: 8s
external_hedge_requests_up_to: 2
external_backend: ""
google_cloud_run: null
external_endpoints: []
trace_by_id:
query_timeout: 10s
max_concurrent_queries: 20
frontend_worker:
frontend_address: 127.0.0.1:9095
dns_lookup_duration: 10s
parallelism: 2
match_max_concurrent: true
id: ""
grpc_client_config:
max_recv_msg_size: 104857600
max_send_msg_size: 16777216
grpc_compression: gzip
rate_limit: 0
rate_limit_burst: 0
backoff_on_ratelimits: false
backoff_config:
min_period: 100ms
max_period: 1s
max_retries: 5
initial_stream_window_size: 0B
initial_connection_window_size: 0B
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
connect_timeout: 0s
connect_backoff_base_delay: 0s
connect_backoff_max_delay: 0s
query_relevant_ingesters: false
query_frontend:
max_outstanding_per_tenant: 2000
querier_forget_delay: 0s
max_batch_size: 5
max_retries: 2
search:
concurrent_jobs: 1000
target_bytes_per_job: 104857600
default_result_limit: 20
max_result_limit: 0
max_duration: 168h0m0s
query_backend_after: 15m0s
query_ingesters_until: 30m0s
trace_by_id:
query_shards: 50
hedge_requests_at: 2s
hedge_requests_up_to: 2
compactor:
ring:
kvstore:
store: ""
prefix: collectors/
consul:
host: localhost:8500
acl_token: ""
http_client_timeout: 20s
consistent_reads: false
watch_rate_limit: 1
watch_burst_size: 1
cas_retry_delay: 1s
etcd:
endpoints: []
dial_timeout: 10s
max_retries: 10
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
username: ""
password: ""
multi:
primary: ""
secondary: ""
mirror_enabled: false
mirror_timeout: 2s
heartbeat_period: 5s
heartbeat_timeout: 1m0s
wait_stability_min_duration: 1m0s
wait_stability_max_duration: 5m0s
instance_id: tempo-0
instance_interface_names:
- eth0
- en0
instance_port: 0
instance_addr: ""
enable_inet6: false
wait_active_instance_timeout: 10m0s
compaction:
v2_in_buffer_bytes: 5242880
v2_out_buffer_bytes: 20971520
v2_prefetch_traces_count: 1000
compaction_window: 1h0m0s
max_compaction_objects: 6000000
max_block_bytes: 107374182400
block_retention: 720h0m0s
compacted_block_retention: 1h0m0s
retention_concurrency: 10
max_time_per_tenant: 5m0s
compaction_cycle: 30s
override_ring_key: compactor
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
prefix: collectors/
consul:
host: localhost:8500
acl_token: ""
http_client_timeout: 20s
consistent_reads: false
watch_rate_limit: 1
watch_burst_size: 1
cas_retry_delay: 1s
etcd:
endpoints: []
dial_timeout: 10s
max_retries: 10
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
username: ""
password: ""
multi:
primary: ""
secondary: ""
mirror_enabled: false
mirror_timeout: 2s
heartbeat_timeout: 5m0s
replication_factor: 1
zone_awareness_enabled: false
excluded_zones: ""
num_tokens: 128
heartbeat_period: 5s
heartbeat_timeout: 1m0s
observe_period: 0s
join_after: 0s
min_ready_duration: 15s
interface_names:
- eth0
enable_inet6: false
final_sleep: 0s
tokens_file_path: ""
availability_zone: ""
unregister_on_shutdown: true
readiness_check_ring_health: true
address: 127.0.0.1
port: 0
id: tempo-0
concurrent_flushes: 4
flush_check_period: 10s
flush_op_timeout: 5m0s
trace_idle_period: 10s
max_block_duration: 30m0s
max_block_bytes: 524288000
complete_block_timeout: 15m0s
override_ring_key: ring
flush_all_on_shutdown: false
metrics_generator:
ring:
kvstore:
store: inmemory
prefix: collectors/
consul:
host: localhost:8500
acl_token: ""
http_client_timeout: 20s
consistent_reads: false
watch_rate_limit: 1
watch_burst_size: 1
cas_retry_delay: 1s
etcd:
endpoints: []
dial_timeout: 10s
max_retries: 10
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
username: ""
password: ""
multi:
primary: ""
secondary: ""
mirror_enabled: false
mirror_timeout: 2s
heartbeat_period: 5s
heartbeat_timeout: 1m0s
instance_id: tempo-0
instance_interface_names:
- eth0
- en0
instance_addr: 127.0.0.1
instance_port: 0
enable_inet6: false
processor:
service_graphs:
wait: 10s
max_items: 10000
workers: 10
histogram_buckets:
- 0.1
- 0.2
- 0.4
- 0.8
- 1.6
- 3.2
- 6.4
- 12.8
dimensions: []
enable_client_server_prefix: false
peer_attributes:
- peer.service
- db.name
- db.system
span_multiplier_key: ""
span_metrics:
histogram_buckets:
- 0.002
- 0.004
- 0.008
- 0.016
- 0.032
- 0.064
- 0.128
- 0.256
- 0.512
- 1.024
- 2.048
- 4.096
- 8.192
- 16.384
intrinsic_dimensions:
service: true
span_name: true
span_kind: true
status_code: true
dimensions: []
dimension_mappings: []
enable_target_info: false
span_multiplier_key: ""
subprocessors:
0: true
1: true
2: true
filter_policies: []
target_info_excluded_dimensions: []
local_blocks:
block:
bloom_filter_false_positive: 0.01
bloom_filter_shard_size_bytes: 102400
version: vParquet2
search_encoding: snappy
search_page_size_bytes: 1048576
v2_index_downsample_bytes: 1048576
v2_index_page_size_bytes: 256000
v2_encoding: zstd
parquet_row_group_size_bytes: 100000000
parquet_dedicated_columns: []
search:
chunk_size_bytes: 1000000
prefetch_trace_count: 1000
read_buffer_count: 32
read_buffer_size_bytes: 1048576
cache_control:
footer: false
column_index: false
offset_index: false
flush_check_period: 10s
trace_idle_period: 10s
max_block_duration: 1m0s
max_block_bytes: 500000000
complete_block_timeout: 1h0m0s
max_live_traces: 0
registry:
collection_interval: 15s
stale_duration: 15m0s
max_label_name_length: 1024
max_label_value_length: 2048
storage:
path: ""
wal:
wal_segment_size: 134217728
wal_compression: none
stripe_size: 16384
truncate_frequency: 2h0m0s
min_wal_time: 300000
max_wal_time: 14400000
no_lockfile: false
remote_write_flush_deadline: 1m0s
remote_write_add_org_id_header: true
traces_storage:
path: ""
completedfilepath: ""
blocksfilepath: ""
v2_encoding: none
search_encoding: none
ingestion_time_range_slack: 0s
version: vParquet2
metrics_ingestion_time_range_slack: 30s
query_timeout: 30s
override_ring_key: metrics-generator
storage:
trace:
pool:
max_workers: 400
queue_depth: 20000
wal:
path: /var/tempo/wal
completedfilepath: /var/tempo/wal/completed
blocksfilepath: /var/tempo/wal/blocks
v2_encoding: snappy
search_encoding: none
ingestion_time_range_slack: 2m0s
version: vParquet2
block:
bloom_filter_false_positive: 0.01
bloom_filter_shard_size_bytes: 102400
version: vParquet2
search_encoding: snappy
search_page_size_bytes: 1048576
v2_index_downsample_bytes: 1048576
v2_index_page_size_bytes: 256000
v2_encoding: zstd
parquet_row_group_size_bytes: 100000000
parquet_dedicated_columns: []
search:
chunk_size_bytes: 1000000
prefetch_trace_count: 1000
read_buffer_count: 32
read_buffer_size_bytes: 1048576
cache_control:
footer: false
column_index: false
offset_index: false
blocklist_poll: 5m0s
blocklist_poll_concurrency: 50
blocklist_poll_fallback: true
blocklist_poll_tenant_index_builders: 2
blocklist_poll_stale_tenant_index: 0s
blocklist_poll_jitter_ms: 0
blocklist_poll_tolerate_consecutive_errors: 1
backend: local
local:
path: /var/tempo/traces
gcs:
bucket_name: ""
prefix: ""
chunk_buffer_size: 10485760
endpoint: ""
hedge_requests_at: 0s
hedge_requests_up_to: 2
insecure: false
object_cache_control: ""
object_metadata: {}
s3:
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: VersionTLS12
bucket: ""
prefix: ""
endpoint: ""
region: ""
access_key: ""
secret_key: ""
session_token: ""
insecure: false
part_size: 0
hedge_requests_at: 0s
hedge_requests_up_to: 2
signature_v2: false
forcepathstyle: false
bucket_lookup_type: 0
tags: {}
storage_class: ""
metadata: {}
native_aws_auth_enabled: false
azure:
storage_account_name: ""
storage_account_key: ""
use_managed_identity: false
use_federated_token: false
user_assigned_id: ""
container_name: ""
prefix: ""
endpoint_suffix: blob.core.windows.net
max_buffers: 4
buffer_size: 3145728
hedge_requests_at: 0s
hedge_requests_up_to: 2
use_v2_sdk: false
cache: ""
cache_min_compaction_level: 0
cache_max_block_age: 0s
background_cache:
writeback_goroutines: 10
writeback_buffer: 10000
memcached: null
redis: null
overrides:
defaults:
ingestion:
rate_strategy: local
rate_limit_bytes: 15000000
burst_size_bytes: 20000000
max_traces_per_user: 10000
read:
max_bytes_per_tag_values_query: 5000000
global:
max_bytes_per_trace: 5000000
per_tenant_override_config: /conf/overrides.yaml
per_tenant_override_period: 10s
user_configurable_overrides:
enabled: false
poll_interval: 1m0s
client:
backend: ""
confirm_versioning: true
local:
path: ""
gcs:
bucket_name: ""
prefix: ""
chunk_buffer_size: 10485760
endpoint: ""
hedge_requests_at: 0s
hedge_requests_up_to: 2
insecure: false
object_cache_control: ""
object_metadata: {}
s3:
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: VersionTLS12
bucket: ""
prefix: ""
endpoint: ""
region: ""
access_key: ""
secret_key: ""
session_token: ""
insecure: false
part_size: 0
hedge_requests_at: 0s
hedge_requests_up_to: 2
signature_v2: false
forcepathstyle: false
bucket_lookup_type: 0
tags: {}
storage_class: ""
metadata: {}
native_aws_auth_enabled: false
azure:
storage_account_name: ""
storage_account_key: ""
use_managed_identity: false
use_federated_token: false
user_assigned_id: ""
container_name: ""
prefix: ""
endpoint_suffix: blob.core.windows.net
max_buffers: 4
buffer_size: 3145728
hedge_requests_at: 0s
hedge_requests_up_to: 2
use_v2_sdk: false
memberlist:
node_name: ""
randomize_node_name: true
stream_timeout: 10s
retransmit_factor: 2
pull_push_interval: 30s
gossip_interval: 1s
gossip_nodes: 2
gossip_to_dead_nodes_time: 30s
dead_node_reclaim_time: 0s
compression_enabled: false
advertise_addr: ""
advertise_port: 7946
cluster_label: ""
cluster_label_verification_disabled: false
join_members: []
min_join_backoff: 1s
max_join_backoff: 1m0s
max_join_retries: 10
abort_if_cluster_join_fails: false
rejoin_interval: 0s
left_ingesters_timeout: 5m0s
leave_timeout: 20s
message_history_buffer_bytes: 0
bind_addr: []
bind_port: 7946
packet_dial_timeout: 2s
packet_write_timeout: 5s
tls_enabled: false
tls_cert_path: ""
tls_key_path: ""
tls_ca_path: ""
tls_server_name: ""
tls_insecure_skip_verify: false
tls_cipher_suites: ""
tls_min_version: ""
usage_report:
reporting_enabled: true
backoff:
min_period: 100ms
max_period: 10s
max_retries: 0

[raagulrakesh]

Hi,
Any update for above?

Thanks in advance

[edgarkz]

you should verify your config and adjust the values, the max trace size works fine, pls try to upgrade your tempo instance to recent 2.5 maybe you have outdated config there