-
Notifications
You must be signed in to change notification settings - Fork 514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tempo doesn't boot. Something about eth0/en0 which don't exist #3590
Comments
Likely related grafana/loki#4483 |
The compactor, as many other Tempo components, uses memberlist to coordinate across instances. As described in the issue I linked, the problem happens when Tempo can't find a default network interface for memberlist. A quick solution is to define it manually in the config compactor:
ring:
instance_addr: 127.0.0.1 Long term, we'll a solution like grafana/loki#4570 |
Could I set the network interface name instead? If so, how? Also, I'm confused as to why this fails / tries to use non-existent network interfaces? |
yes, you can configure custom interface names to check for the ip address. there are few ring configs: distributor, ingester, compactor, and metrics generator that will need this change. you can search the manifest for https://grafana.com/docs/tempo/latest/configuration/manifest/ |
Switched to: server:
http_listen_port: 3200
grpc_listen_port: 9095
querier:
frontend_worker:
frontend_address: xi.xmit.dev:9095
query_frontend:
search:
duration_slo: 5s
trace_by_id:
duration_slo: 5s
distributor:
ring: &1
instance_interface_names:
- enp7s0
kvstore:
store: etcd
etcd:
endpoints:
- 127.0.0.1:2379
tls_enabled: true
tls_cert_path: /etc/ssl/etcd/cert.pem
tls_key_path: /etc/ssl/etcd/key.pem
tls_ca_path: /etc/ssl/etcd/ca.pem
receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:4317
ingester:
lifecycler:
ring: *1
max_block_duration: 5m
compactor:
ring: *1
compaction:
block_retention: 1h
metrics_generator:
ring: *1
storage:
path: /srv/tempo/generator/wal
remote_write:
- url: http://127.0.0.1:9090/api/v1/write
send_exemplars: true
storage:
trace:
backend: s3
s3:
bucket: tempo
endpoint: 127.0.0.1:9000
insecure: true
access_key: CENSORED
secret_key: CENSORED
wal:
path: /srv/tempo/wal
v2_encoding: snappy
block:
v2_encoding: zstd
overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics] Fails with:
|
The ring config in the ingester is slightly different than in the distributor or compactor. Try with this ingester:
lifecycler:
interface_names:
- enp7s0
ring:
kvstore:
store: etcd
etcd:
...
max_block_duration: 5m |
Ah, now looks like grafana-server tries to connect to :3200 over IPv6 but tempo only listens on IPv4… Does that sound about right? |
Tempo can operate using IPv4 or IPv6, but the ring can use only one at a time. See our docs on IPv6 if that is your intent. https://grafana.com/docs/tempo/latest/configuration/network/ipv6/ |
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. |
Note that Tempo wouldn't deploy on Railway because of this as their network interface is named |
As long as the interface has private addresses on it, it should be detected. May I see the |
Describe the bug
tempo doesn't start, logs:
To Reproduce
Steps to reproduce the behavior:
Expected behavior
tempo starts successfully.
Environment:
Additional Context
The text was updated successfully, but these errors were encountered: