observ is a minimal go application reporting RED and USE prometheus metrics
RED metrics include:
- (Requests) request count and rate
- (Errors) HTTP error count and rate
- (Duration) HTTP request/response time
USE measurements include:
- (Utilization) Percentage of workers that are busy
- (Saturation) Size of the work queue
- (Errors) Number of jobs that ended in error
RED metrics deal with requests and are generated automatically with prometheus.InstrumentHander
USE metrics deal with resources like CPU, MEM, disk, or in our case ... workers!
worker metrics are custom metrics and are instrumented using prometheus.Gauge and prometheus.Counter
Note: default worker count and queue length is 4
docker-compose up
observ reports prometheus metrics at /metrics
after the first request:
curl -s localhost:8111/req && curl -s localhost:8111/metrics
These ones are the RED Metrics:
http_request_duration_microseconds_sum{handler="/req"} 15.9
http_request_duration_microseconds_count{handler="/req"} 1
http_requests_total{code="200",handler="/req",method="get"} 1
and these are the USE metrics:
observ_worker_errors 0
observ_worker_saturation 0
observ_worker_utilization 0
GET
/req
simulates http request traffic with the following params:
param | units | default | description |
---|---|---|---|
duration | millisecs | 0 | ms delay before HTTP response |
httpcode | int | 200 | http code of response |
worksecs | seconds | 0 | seconds of work to add to queue |
workfail | bool | false | set true to simulate work failing |
Simulate a request that takes 472ms to return 302 but results in no added work
curl -si "localhost:8111/req?duration=472&httpcode=302"
Simulate a request that returns instantly but results in 6 seconds of added work.
curl -si "localhost:8111/req?worksecs=6"
Add a requtest that takes 150ms to return HTTP 201, and produces 6 seconds of work that ultimately fails.
curl -si "localhost:8111/req?duration=150&httpcode=201&worksecs=6&workfail=true"
Saturate workqueue (Note: responds HTTP 507)
while true; do curl -si localhost:8111/req?worksecs=11; sleep 1; done
Use localhost:8111/metrics
or the links below to verify prometheus metrics:
-
Request rates go up the more often
/req
endpoint is hit -
Errors go up the more the
httpcode
param is set to 5xx -
Duration goes up when
duration
is set higher -
Utilization (number of workers in use) and,
-
Saturation (number of jobs in queue) go up with work and work is created with each non-zero
worksecs
request. -
Error (count) increases each
workfail=true
- CD
- Serverless version to simulate job metrics
- Grafana