Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make capture interval a multiple of 30s #187

Merged
merged 1 commit into from
Dec 14, 2017
Merged

Make capture interval a multiple of 30s #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion app/models/manageiq/providers/kubernetes/container_manager/metrics_capture.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def log_severity
require_nested :HawkularCaptureContext
require_nested :PrometheusCaptureContext

INTERVAL = 20.seconds
INTERVAL = 60.seconds

VIM_STYLE_COUNTERS = {
"cpu_usage_rate_average" => {
Expand Down Expand Up @@ -51,6 +51,9 @@ def log_severity
}

def capture_context(ems, target, start_time, end_time)
# make start_time align to minutes
start_time = start_time.beginning_of_minute

# check for prometheus endpoint, ems must be set
if ems.connection_configurations.prometheus.try(:endpoint)
PrometheusCaptureContext.new(target, start_time, end_time, INTERVAL)
Expand Down
29 changes: 20 additions & 9 deletions .../manageiq/providers/kubernetes/container_manager/metrics_capture/capture_context_mixin.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,11 @@ def fetch_counters_rate(resource)

def process_cpu_counters_rate(counters_rate)
@metrics |= ['cpu_usage_rate_average'] unless counters_rate.empty?
total_cpu_time = @node_cores * CPU_NANOSECONDS * @interval
sec_cpu_time = @node_cores * CPU_NANOSECONDS
counters_rate.each do |x|
interval = (x['end'] - x['start']) / 1.in_milliseconds
timestamp = Time.at(x['start'] / 1.in_milliseconds).utc
avg_usage = (x['avg'] * 100.0) / total_cpu_time
avg_usage = (x['min'] * 100.0) / (sec_cpu_time * interval)
@ts_values[timestamp]['cpu_usage_rate_average'] = avg_usage
end
end
Expand All @@ -72,16 +73,17 @@ def process_mem_gauges_data(gauges_data)
@metrics |= ['mem_usage_absolute_average'] unless gauges_data.empty?
gauges_data.each do |x|
timestamp = Time.at(x['start'] / 1.in_milliseconds).utc
avg_usage = (x['avg'] / 1.megabytes) * 100.0 / @node_memory
avg_usage = (x['min'] / 1.megabytes) * 100.0 / @node_memory
@ts_values[timestamp]['mem_usage_absolute_average'] = avg_usage
end
end

def process_net_counters_rate(counters_rate)
@metrics |= ['net_usage_rate_average'] unless counters_rate.empty?
counters_rate.each do |x|
interval = (x['end'] - x['start']) / 1.in_milliseconds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, this is true only if n['start'] the same as prv['end'], right?

otherwise we would need to move the calculation to below, where we have n and prv elements available?

Copy link
Member Author

@yaacov yaacov Dec 11, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ladas 👍 fixed, now x['end'] is defined as next['start'] on line 114.

timestamp = Time.at(x['start'] / 1.in_milliseconds).utc
avg_usage_kb = x['avg'] / (1.kilobyte.to_f * @interval)
avg_usage_kb = x['min'] / (1.kilobyte.to_f * interval)
@ts_values[timestamp]['net_usage_rate_average'] = avg_usage_kb
end
end
Expand All @@ -98,19 +100,28 @@ def compute_summation(data)
{
'start' => k,
'end' => [sum['end'], n['end']].max,
'avg' => sum['avg'] + n['avg']
'min' => sum['min'] + n['min']
}
end
end
end

def compute_derivative(counters)
counters.each_cons(2).map do |prv, n|
# Add min, median, max, percentile95th, etc. if needed
# Add min, median, min, percentile95th, etc. if needed
# time window:
# 00:00 01:00
# ^ (sample start time) ^ (next sample start time)
# ^ (sample min/max/avg value) ^ (next sample min/max/avg value)
# ^ (real sample time) ^ (real sample time) ^ (real sample time)
# ^ (real sample value) ^ (real sample value) ^ (real sample value)
# we use:
# (T = start of window timestamp, V = min value of window samples)
# because the min value is the value of the sample closest to start of window.
{
'start' => n['start'],
'end' => n['end'],
'avg' => n['avg'] - prv['avg']
'start' => prv['start'],
'end' => n['start'],
'min' => n['min'] - prv['min']
}
end
end
Expand Down
2 changes: 1 addition & 1 deletion ...nageiq/providers/kubernetes/container_manager/metrics_capture/hawkular_capture_context.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ def insert_metrics_key(raw_metrics, key, full_key)
# insert the raw metrics into the @ts_values global object
raw_metrics['gauge'][full_key].each do |metric|
timestamp = Time.at(metric['start'] / 1.in_milliseconds).utc
@ts_values[timestamp][key] = metric['avg'] unless metric['empty']
@ts_values[timestamp][key] = metric['max'] unless metric['empty']
end
end

Expand Down
3 changes: 2 additions & 1 deletion ...geiq/providers/kubernetes/container_manager/metrics_capture/prometheus_capture_context.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ def sort_and_normalize(response)

{
"start" => start_sec.to_i.in_milliseconds,
"avg" => x[1].to_f
"end" => (start_sec.to_i + @interval.to_i).in_milliseconds,
"min" => x[1].to_f
}
end
end
Expand Down
3 changes: 2 additions & 1 deletion ...providers/kubernetes/container_manager/metrics_capture/prometheus_capture_context_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@

data = context.collect_metrics

expect(data.count).to be < 18
expect(data.count).to be > 10
expect(data.count).to be < 13
end
end
end
Expand Down
66 changes: 33 additions & 33 deletions spec/models/manageiq/providers/kubernetes/container_manager/metrics_capture_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,85 +47,85 @@
# TODO: include also sort_and_normalize in the tests
METRICS_EXERCISES = [
{
:counters => [
:counters => [
{
:args => 'cpu/usage',
:data => [
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 0},
{'start' => 1446500020000, 'end' => 1446500040000, 'avg' => 4000000000}
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 0},
{'start' => 1_446_500_060_000, 'end' => 1_446_500_120_000, 'min' => 12_000_000_000},
]
},
{
:args => 'network/tx',
:data => [
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 0},
{'start' => 1446500020000, 'end' => 1446500040000, 'avg' => 153600}
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 0},
{'start' => 1_446_500_060_000, 'end' => 1_446_500_120_000, 'min' => 460_800}
]
},
{
:args => 'network/rx',
:data => [
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 0},
{'start' => 1446500020000, 'end' => 1446500040000, 'avg' => 51200}
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 0},
{'start' => 1_446_500_060_000, 'end' => 1_446_500_120_000, 'min' => 153_600}
]
}
],
:gauges => [
:gauges => [
{
:args => 'memory/usage',
:data => [
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 1073741824}
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 1_073_741_824}
]
}
],
:node_expected => {},
:container_expected => {}
:node_expected => {
Time.at(1_446_500_000).utc => {
"cpu_usage_rate_average" => 10.0,
"mem_usage_absolute_average" => 50.0,
"net_usage_rate_average" => 10.0
}
},
:container_expected => {
Time.at(1_446_500_000).utc => {
"cpu_usage_rate_average" => 10.0,
"mem_usage_absolute_average" => 50.0
}
}
},
{
:counters => [
:counters => [
{
:args => 'cpu/usage',
:data => [
{'start' => 1446499980000, 'end' => 1446500000000, 'avg' => 0},
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 4000000000}
{'start' => 1_446_499_940_000, 'end' => 1_446_500_000_000, 'min' => 0},
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 12_000_000_000}
]
},
{
:args => 'network/tx',
:data => [
{'start' => 1446499980000, 'end' => 1446500000000, 'avg' => 0},
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 153600}
{'start' => 1_446_499_940_000, 'end' => 1_446_500_000_000, 'min' => 0},
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 460_800}
]
},
{
:args => 'network/rx',
:data => [
{'start' => 1446499980000, 'end' => 1446500000000, 'avg' => 0},
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 51200}
{'start' => 1_446_499_940_000, 'end' => 1_446_500_000_000, 'min' => 0},
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 153_600}
]
}
],
:gauges => [
:gauges => [
{
:args => 'memory/usage',
:data => [
{'start' => 1446500000000, 'end' => 1446500020000, 'avg' => 1073741824}
{'start' => 1_446_500_000_000, 'end' => 1_446_500_060_000, 'min' => 1_073_741_824}
]
}
],
:node_expected => {
Time.at(1_446_500_000).utc => {
"cpu_usage_rate_average" => 10.0,
"mem_usage_absolute_average" => 50.0,
"net_usage_rate_average" => 10.0
}
},
:container_expected => {
Time.at(1_446_500_000).utc => {
"cpu_usage_rate_average" => 10.0,
"mem_usage_absolute_average" => 50.0
}
}
:node_expected => {},
:container_expected => {}
}
]

Expand Down