Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[nginx] Metrics missing after updating to nginx-0.9.0-beta.12 #1279

Closed
danielfm opened this issue Aug 31, 2017 · 4 comments · Fixed by #1282
Closed

[nginx] Metrics missing after updating to nginx-0.9.0-beta.12 #1279

danielfm opened this issue Aug 31, 2017 · 4 comments · Fixed by #1282

Comments

@danielfm
Copy link

After updating nginx ingress from 0.9.0-beta.11 to beta.12, I noticed nginx_* metrics were gone. Reverting to beta.11 restored the metrics.

New version:

$ kubectl -n kube-system exec hissing-wasp-nginx-ingress-controller-1849821676-xq99c   -- curl http://localhost:10254/metrics
Defaulting container name to nginx-ingress-controller.
Use 'kubectl describe pod/hissing-wasp-nginx-ingress-controller-1849821676-xq99c' to see all of the containers in this pod.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12825  100 12825    0     0  1769# HELP cpu_seconds_total Cpu usage in seconds
# TYPE cpu_seconds_total counter
cpu_seconds_total 0.55
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.5419e-05
go_gc_duration_seconds{quantile="0.25"} 7.842e-05
go_gc_duration_seconds{quantile="0.5"} 0.000198881
go_gc_duration_seconds{quantile="0.75"} 0.004966737
go_gc_duration_seconds{quantile="1"} 0.008900403
go_gc_duration_seconds_sum 0.02436825
go_gc_duration_seconds_count 11
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 68
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.8829968e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 8.4096536e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.463967e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 333855
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.00345517148834525
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 1.968128e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.8829968e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 3.0253056e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.1061632e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 92296
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.1314688e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.504185879913565e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 508
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 426151
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 239400
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 409600
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 3.2266192e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 649817
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.114112e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.114112e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 5.6936696e+07
# HELP go_threads Number of OS threads created
# TYPE go_threads gauge
go_threads 11
# HELP ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire. An example to check if this certificate will expire in 10 days is: "ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))"
# TYPE ingress_controller_ssl_expire_time_seconds gauge
ingress_controller_ssl_expire_time_seconds{host="alertmanager-k8s-1.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="alertmanager-k8s-1.prd.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="alertmanager-k8s-2.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="alertmanager-k8s-2.prd.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-1063.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-1111.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-1115.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-1116.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-1117.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera-945.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="chimera.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="desauth.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="gabarito.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="grafana-k8s.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="grafana-k8s.prd.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-267.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-279.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-285.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-291.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-333.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-344.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-350.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-368.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-373.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-380.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-383.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-384.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-399.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings-435.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="landings.stg.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1124-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1124.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1141.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1173-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1173.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1197-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1197.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1219-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1219.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1238-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1238.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1241-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1241.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1242-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1242.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1244-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1244.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1245-worker.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-1245.dev.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap-worker.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="lap.stg.descomplica.com.br"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="notification-queue.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="partitus.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="passepartout-upstream.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="pepper.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="prometheus-k8s-1.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="prometheus-k8s-1.prd.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="prometheus-k8s-2.dev.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="prometheus-k8s-2.prd.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="quaestio.stg.descomplica.io"} -6.21355968e+10
ingress_controller_ssl_expire_time_seconds{host="sisu.stg.descomplica.com.br"} -6.21355968e+10
# HELP ingress_controller_success Cumulative number of Ingress controller reload operations
# TYPE ingress_controller_success counter
ingress_controller_success{count="reloads"} 1
# HELP num_procs number of processes
# TYPE num_procs gauge
num_procs 4
# HELP oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE oldest_start_time_seconds gauge
oldest_start_time_seconds 1.504185857e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.67
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 18
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 7.15776e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.50418585775e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 8.4688896e+07
# HELP read_bytes_total number of bytes read
# TYPE read_bytes_total counter
read_bytes_total 0
# HELP resident_memory_bytes number of bytes of memory in use
# TYPE resident_memory_bytes gauge
resident_memory_bytes 1.0952704e+08
# HELP virtual_memory_bytes number of bytes of memory in use
# TYPE virtual_memory_bytes gauge
virtual_memory_bytes 3.40094976e+08
# HELP write_bytes_total number of bytes written
# TYPE write_bytes_total counter
write_bytes_total 1.212416e+06

Old version:

$ kubectl -n kube-system exec altered-echidna-nginx-ingress-controller-3109702695-63kc2 -- curl http://localhost:10254/metrics
Defaulting container name to nginx-ingress-controller.
Use 'kubectl describe pod/altered-echidna-nginx-ingress-controller-3109702695-63kc2' to see all of the containers in this pod.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 25282  100 25282    0     0  2934k      0 --:--:-- --:--:-- --:--:-- 3086k
# HELP cpu_seconds_total Cpu usage in seconds
# TYPE cpu_seconds_total counter
cpu_seconds_total 0.59
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.1256e-05
go_gc_duration_seconds{quantile="0.25"} 7.9157e-05
go_gc_duration_seconds{quantile="0.5"} 9.6377e-05
go_gc_duration_seconds{quantile="0.75"} 0.000173161
go_gc_duration_seconds{quantile="1"} 0.005364642
go_gc_duration_seconds_sum 0.007494909
go_gc_duration_seconds_count 13
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 69
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.6374648e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 7.762408e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.462823e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 260612
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.001283637985066354
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 1.599488e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.6374648e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.3355392e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.8620416e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 66510
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 4.1975808e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.504185426064754e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 278
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 327122
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 210216
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 278528
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 3.129912e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 519889
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.015808e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.015808e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 4.6868728e+07
# HELP go_threads Number of OS threads created
# TYPE go_threads gauge
go_threads 11
# HELP ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire. An example to check if this certificate will expire in 10 days is: "ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))"
# TYPE ingress_controller_ssl_expire_time_seconds gauge
ingress_controller_ssl_expire_time_seconds{host="livews.stg.descomplica.com.br"} -6.21355968e+10
# HELP ingress_controller_success Cumulative number of Ingress controller reload operations
# TYPE ingress_controller_success counter
ingress_controller_success{count="reloads"} 1
# HELP nginx_bytes_total Nginx bytes count
# TYPE nginx_bytes_total counter
nginx_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*"} 692
nginx_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_"} 692
nginx_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*"} 2438
nginx_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_"} 2438
# HELP nginx_cache_total Nginx cache count
# TYPE nginx_cache_total counter
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="bypass"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="expired"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="hit"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="miss"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="revalidated"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="scarce"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="stale"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",type="updating"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="bypass"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="expired"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="hit"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="miss"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="revalidated"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="scarce"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="stale"} 0
nginx_cache_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",type="updating"} 0
# HELP nginx_connections_total Nginx connections count
# TYPE nginx_connections_total counter
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="accepted"} 48
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="active"} 1
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="handled"} 48
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="reading"} 0
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="requests"} 7
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="waiting"} 0
nginx_connections_total{ingress_class="nginx-ingress-stg-ws",namespace="",type="writing"} 1
# HELP nginx_filterzone_bytes_total Nginx bytes count
# TYPE nginx_filterzone_bytes_total counter
nginx_filterzone_bytes_total{country="CN",direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*"} 143
nginx_filterzone_bytes_total{country="CN",direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_"} 143
nginx_filterzone_bytes_total{country="CN",direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*"} 235
nginx_filterzone_bytes_total{country="CN",direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_"} 235
# HELP nginx_filterzone_cache_total Nginx cache count
# TYPE nginx_filterzone_cache_total counter
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="bypass"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="expired"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="hit"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="miss"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="revalidated"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="scarce"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="stale"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",type="updating"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="bypass"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="expired"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="hit"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="miss"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="revalidated"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="scarce"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="stale"} 0
nginx_filterzone_cache_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",type="updating"} 0
# HELP nginx_filterzone_responses_total The number of responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.
# TYPE nginx_filterzone_responses_total counter
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",status_code="1xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",status_code="2xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",status_code="3xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",status_code="4xx"} 1
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::*",status_code="5xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",status_code="1xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",status_code="2xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",status_code="3xx"} 0
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",status_code="4xx"} 1
nginx_filterzone_responses_total{country="CN",ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="country::_",status_code="5xx"} 0
# HELP nginx_requests_total The total number of requested client connections.
# TYPE nginx_requests_total counter
nginx_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*"} 6
nginx_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_"} 6
# HELP nginx_responses_total The number of responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.
# TYPE nginx_responses_total counter
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",status_code="1xx"} 0
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",status_code="2xx"} 5
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",status_code="3xx"} 0
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",status_code="4xx"} 1
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="*",status_code="5xx"} 0
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",status_code="1xx"} 0
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",status_code="2xx"} 5
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",status_code="3xx"} 0
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",status_code="4xx"} 1
nginx_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server_zone="_",status_code="5xx"} 0
# HELP nginx_upstream_backup Current backup setting of the server.
# TYPE nginx_upstream_backup counter
nginx_upstream_backup{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 0
nginx_upstream_backup{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_backup{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_backup{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_bytes_total The total number of bytes sent to this server.
# TYPE nginx_upstream_bytes_total counter
nginx_upstream_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 143
nginx_upstream_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_bytes_total{direction="in",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 235
nginx_upstream_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_bytes_total{direction="out",ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_fail_timeout Current fail_timeout setting of the server.
# TYPE nginx_upstream_fail_timeout counter
nginx_upstream_fail_timeout{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 0
nginx_upstream_fail_timeout{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_fail_timeout{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_fail_timeout{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_maxfails Current max_fails setting of the server.
# TYPE nginx_upstream_maxfails counter
nginx_upstream_maxfails{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 0
nginx_upstream_maxfails{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_maxfails{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_maxfails{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_requests_total The total number of client connections forwarded to this server.
# TYPE nginx_upstream_requests_total counter
nginx_upstream_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 1
nginx_upstream_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_requests_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_response_msecs_avg The average of only upstream response processing times in milliseconds.
# TYPE nginx_upstream_response_msecs_avg counter
nginx_upstream_response_msecs_avg{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 0
nginx_upstream_response_msecs_avg{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_upstream_response_msecs_avg{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_upstream_response_msecs_avg{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_responses_total The number of upstream responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.
# TYPE nginx_upstream_responses_total counter
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",status_code="1xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",status_code="2xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",status_code="3xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",status_code="4xx",upstream="upstream-default-backend"} 1
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",status_code="5xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",status_code="1xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",status_code="2xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",status_code="3xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",status_code="4xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",status_code="5xx",upstream="upstream-default-backend"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",status_code="1xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",status_code="2xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",status_code="3xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",status_code="4xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",status_code="5xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",status_code="1xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",status_code="2xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",status_code="3xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",status_code="4xx",upstream="staging-live-staging-80"} 0
nginx_upstream_responses_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",status_code="5xx",upstream="staging-live-staging-80"} 0
# HELP nginx_upstream_weight Current upstream weight setting of the server.
# TYPE nginx_upstream_weight counter
nginx_upstream_weight{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 1
nginx_upstream_weight{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 1
nginx_upstream_weight{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 1
nginx_upstream_weight{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 1
# HELP nginx_vts_upstream_down_total Current down setting of the server.
# TYPE nginx_vts_upstream_down_total counter
nginx_vts_upstream_down_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.103.5:8080",upstream="upstream-default-backend"} 0
nginx_vts_upstream_down_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.12.8:8080",upstream="upstream-default-backend"} 0
nginx_vts_upstream_down_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.31.23:5000",upstream="staging-live-staging-80"} 0
nginx_vts_upstream_down_total{ingress_class="nginx-ingress-stg-ws",namespace="",server="10.2.79.7:5000",upstream="staging-live-staging-80"} 0
# HELP num_procs number of processes
# TYPE num_procs gauge
num_procs 4
# HELP oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE oldest_start_time_seconds gauge
oldest_start_time_seconds 1.5041854e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.72
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 16
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.201344e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.50418540064e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.4305536e+07
# HELP read_bytes_total number of bytes read
# TYPE read_bytes_total counter
read_bytes_total 0
# HELP resident_memory_bytes number of bytes of memory in use
# TYPE resident_memory_bytes gauge
resident_memory_bytes 9.6796672e+07
# HELP virtual_memory_bytes number of bytes of memory in use
# TYPE virtual_memory_bytes gauge
virtual_memory_bytes 3.32296192e+08
# HELP write_bytes_total number of bytes written
# TYPE write_bytes_total counter
write_bytes_total 73728

Did anything change in the way nginx_* metrics were collected and reported between these versions?
@danielfm
Copy link
Author

danielfm commented Aug 31, 2017 •
edited
Loading

Found lots of errors like this in the nginx controller logs:

W0831 13:29:06.060158       5 vts.go:206] unexpected error obtaining nginx status info: unexpected error scraping nginx vts (unexpected error scraping nginx : Get http://localhost:0/nginx_status/format/json: dial tcp [::1]:0: getsockopt: connection refused)

It seems the port vts port is set to zero, instead of 18080, not sure why. I'll dig a little more, it seems like a misconfiguration on my side.

@danielfm
Copy link
Author

danielfm commented Aug 31, 2017 •
edited
Loading

The status port is initialized here:

https://github.com/kubernetes/ingress/blob/2d6f811151c3829b42b175745f7f6a9d0fda38ea/controllers/nginx/pkg/cmd/controller/nginx.go#L338-L345

But the code passes the supposedly uninitialized (zero-valued) n.ports.Health to the collector, instead of what I suppose should be n.ports.Status:

https://github.com/kubernetes/ingress/blob/2d6f811151c3829b42b175745f7f6a9d0fda38ea/controllers/nginx/pkg/cmd/controller/nginx.go#L375

@danielfm danielfm mentioned this issue Aug 31, 2017
Closed
@aledbf
Copy link
Member

aledbf commented Aug 31, 2017

Please test quay.io/aledbf/nginx-ingress-controller:0.202

@aledbf aledbf mentioned this issue Aug 31, 2017
Merged
@danielfm
Copy link
Author

@aledbf it seems this version seems to work. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix nginx stats aledbf/ingress-nginx
2 participants
@danielfm @aledbf