Skip to content

Commit

Permalink
v2: Ditch Query helpers.
Browse files Browse the repository at this point in the history 
This PR focuses around simplification of the tsplot package. Moving away
from something that was rather perscriptive in the type of Query that
you could make and towards a package that can accept a Time Series or
Time Series Itterator from the Google Metrics API and return a rendered
graph of those Time Series.
  • Loading branch information
@jharshman
jharshman committed Dec 11, 2023
1 parent b97b9fd commit 739d6af
Show file tree
Hide file tree
Showing 11 changed files with 97 additions and 655 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
.idea/*
.DS_store
14 changes: 1 addition & 13 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,9 @@ test:
$(GOBIN) test -v ./...

.PHONY: build
build: $(BINDIR)/codegen
build:
make -C tscli

$(BINDIR)/codegen: $(SCRIPTDIR)/codegen.go
@mkdir -p $(BINDIR)
GOOS=linux GOARCH=amd64 $(GOBIN) build -o $(BINDIR)/codegen $<

.PHONY: install
install: build
cp $(BINDIR)/tscli /usr/local/bin/
Expand All @@ -26,11 +22,3 @@ install: build
clean:
rm -rf $(BINDIR)

CODEGENFILE="set_aggregation_opts.go"
.PHONY: codegen
codegen:
$(BINDIR)/codegen -output ./tsplot/$(CODEGENFILE)

.PHONY: codegendiff
codegendiff:
diff ./tsplot/$(CODEGENFILE) <($(BINDIR)/codegen -stdout)
82 changes: 21 additions & 61 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,84 +2,44 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/bitly/tsplot)](https://goreportcard.com/report/github.com/bitly/tsplot)
[![Go Reference](https://pkg.go.dev/badge/github.com/bitly/tsplot.svg)](https://pkg.go.dev/github.com/bitly/tsplot)

This package provides a method of querying for raw time series data from the GCM APIs and additionally plotting that data for use in other applications.

This came to be due to what we consider a small limitation in the Google APIs which require us to re-draw graphs to include them in other applications such as
Slack bots. There is no facility in the Google API that provides a PNG of already graphed data.

## Authentication
This package makes no effort to assist in authentication to the Google APIs.
Instead, it will expect the caller to supply an authenticated client.

More information on authentication can be found in the official [Google Cloud documentation](https://cloud.google.com/docs/authentication).

## Query
tsplot helps to facilitate easy querying of the Google Cloud Monitoring API for time series matching the supplied criteria.
In addition it provides methods of overriding certain aspects of the query.

For example, the following code snippet will return a single time series for the following metric descriptor: `custom.googleapis.com/opencensus/fishnet/queuereader_fishnet/messages_total`.
```
func main() {
... snip ...
... snip ...
start := time.Now().Add(-1 * time.Hour)
end := time.Now()
mq := &tsplot.NewMetricQuery(
"bitly-gcp-prod", // GCP project
"custom.googleapis.com/opencensus/fishent/queuereader_fishnet/messages_total", // metric descriptor
&start, // start of time window
&end, // end of time window
)
// disable cross series reducer (MEAN reduction is default)
query.Set_REDUCE_NONE()
// set different alignment window. (Default is 1 minute)
query.SetAlignmentPeriod(time.Minute * 2)
tsi, err := mq.PerformWithClient(client) // client is provided by user
if err != nil {
fmt.Printf("error performing query: %v\n", err)
}
}
```

## Plotting
To plot the data, tsplot leverages the open source package [gonum/plot](github.com/gonum/plot) to create a graph and plot the data for a given time series.

The example below creates a new graph containing a singular time series, plots it, and saves the resulting plot to disk.
```
func main() {
... snip ...
ts := tsplot.TimeSeries{}
// optionally iterate over returned time series
timeSeries, _ := tsi.Next()
ts[metric] = ts.GetPoints()
// create the plot with some formatting options
p, err := ts.Plot([]tsplot.PlotOption{
tsplot.WithXAxisName("UTC"),
tsplot.WIthGrid(colornames.Darkgrey),
tsplot.WithTitle(metric)}...)
if err != nil {
return err
// create new request
request := monitoringpb.ListTimeSeriesRequest{
Name: fmt.Sprintf("projects/%s", project),
Filter: query,
Interval: &monitoringpb.TimeInterval{
EndTime: timestamppb.New(et),
StartTime: timestamppb.New(st),
},
Aggregation: nil,
SecondaryAggregation: nil,
View: monitoringpb.ListTimeSeriesRequest_FULL,
}
// optionally save the plot to disk
p.Save(8*vg.Inch, 4*vg.Inch, "./my-graph.png")
// execute the request and get the response from Google APIs
tsi := GoogleCloudMonitoringClient.ListTimeSeries(context.Background(), request)
// Create the plot from the GAPI TimeSeries
plot, _ := tsplot.NewPlotFromTimeSeriesIterator(tsi)
// Save the new plot to disk.
plot.Save(8*vg.Inch, 4*vg.Inch, "my_plot.png")
}
```

### Example generated graphs:
Query across multiple time series with mean reducer:
![graph1](sample/1.png)

### Graph Color Scheme
I'm not a UX designer, but I have selected colors that I find higher contrast
and easier to see. I am basing this completely off my colorblindness which is
unique to me. Improvements to the color palette used are welcome.
67 changes: 0 additions & 67 deletions scripts/codegen.go
View file

This file was deleted.

2 changes: 1 addition & 1 deletion tscli/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ default: build
.PHONY: build
build:
@mkdir -p $(BINDIR)
GOOS=linux GOARCH=amd64 $(GOBIN) build -o $(BINDIR)/tscli
$(GOBIN) build -o $(BINDIR)/tscli
121 changes: 36 additions & 85 deletions tscli/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,21 @@ import (
"context"
"errors"
"fmt"
"github.com/bitly/tsplot/tsplot"
"gonum.org/v1/plot/vg"
"google.golang.org/protobuf/types/known/durationpb"
"google.golang.org/protobuf/types/known/timestamppb"
"log"
"os"
"regexp"
"strconv"
"strings"
"time"

monitoring "cloud.google.com/go/monitoring/apiv3/v2"
"github.com/bitly/tsplot/tsplot"
"github.com/spf13/cobra"
"golang.org/x/image/colornames"
"gonum.org/v1/plot/vg"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
monitoringpb "google.golang.org/genproto/googleapis/monitoring/v3"
)

const GAP = "GOOGLE_APPLICATION_CREDENTIALS"
Expand Down Expand Up @@ -48,33 +50,21 @@ from Google Cloud Monitoring (formerly StackDriver).
RunE: executeQuery,
}

project string
app string
service string
metric string
startTime string
endTime string
queryOverride string
outDir string
reduce bool
justPrint bool
project string
query string
startTime string
endTime string
outDir string
)

func init() {
rootCmd.Flags().StringVarP(&project, "project", "p", "", "GCP Project.")
rootCmd.Flags().StringVarP(&app, "app", "a", "", "The (Bitly) application. Usually top level directory")
rootCmd.Flags().StringVarP(&service, "service", "s", "", "The (Bitly) service. Service directory found under application directory.")
rootCmd.Flags().StringVarP(&metric, "metric", "m", "", "The metric.")
rootCmd.Flags().StringVarP(&query, "query", "m", "", "The query filter.")
rootCmd.Flags().StringVar(&startTime, "start", "", "Start time of window for which the query returns time series data for. Hours or minutes accepted, i.e: -5h or -5m.")
rootCmd.Flags().StringVar(&endTime, "end", "now", "End of the time window for which the query returns time series data for. Hours or minutes accepted, i.e: -5h or -5m or now.")
rootCmd.Flags().BoolVar(&justPrint, "print-raw", false, "Only print time series data and exit.")
rootCmd.Flags().BoolVar(&reduce, "reduce", false, "Use a time series reducer to return a single averaged result.")
rootCmd.Flags().StringVar(&queryOverride, "query-override", "", "Override the default query. Must be a full valid query. Metric flag is not used.")
rootCmd.Flags().StringVarP(&outDir, "output", "o", "", "Specify output directory for resulting plot. Defaults to current working directory.")
rootCmd.MarkFlagRequired("project")
rootCmd.MarkFlagRequired("app")
rootCmd.MarkFlagRequired("service")
rootCmd.MarkFlagRequired("metric")
//rootCmd.MarkFlagRequired("query")
rootCmd.MarkFlagRequired("start")
}

Expand All @@ -95,10 +85,6 @@ func auth(cmd *cobra.Command, args []string) error {

func executeQuery(cmd *cobra.Command, args []string) error {

if metric != "" && queryOverride != "" {
fmt.Println("warn: both --metric and --query-override flag used. Favoring --query-override.")
}

if !timeFormatOK(startTime) {
return errors.New("err validating start time format")
}
Expand All @@ -114,69 +100,34 @@ func executeQuery(cmd *cobra.Command, args []string) error {
st := parseTime(startTime)
et := parseTime(endTime)

query := tsplot.NewMetricQuery(
project,
fmt.Sprintf("custom.googleapis.com/opencensus/%s/%s/%s", app, service, metric),
&st,
&et,
)

if queryOverride != "" {
query.SetQueryFilter(queryOverride)
}

if !reduce {
query.Set_REDUCE_NONE()
}

tsi, err := query.PerformWithClient(GoogleCloudMonitoringClient)
request := &monitoringpb.ListTimeSeriesRequest{
Name: fmt.Sprintf("projects/%s", project),
Filter: `resource.type = "global" AND metric.type = "custom.googleapis.com/opencensus/fishnet/queuereader_fishnet/messages_total"`,
Interval: &monitoringpb.TimeInterval{
EndTime: timestamppb.New(et),
StartTime: timestamppb.New(st),
},
Aggregation: &monitoringpb.Aggregation{
AlignmentPeriod: durationpb.New(time.Minute * 1),
PerSeriesAligner: monitoringpb.Aggregation_ALIGN_RATE,
CrossSeriesReducer: monitoringpb.Aggregation_REDUCE_MEAN,
},
View: monitoringpb.ListTimeSeriesRequest_FULL,
}

tsi := GoogleCloudMonitoringClient.ListTimeSeries(context.Background(), request)
// todo: handle multiple time series
ts, err := tsi.Next()
if err != nil {
return err
log.Fatal(err)
}

ts := tsplot.TimeSeries{}
for {
timeSeries, err := tsi.Next()
if err != nil {
if err == iterator.Done {
break
}
return err
}

// todo: implement "just-print" mode for multiple time series
//if justPrint {
// fmt.Printf("%v", timeSeries)
// return nil
//}

// key helps to fill out legend.
// Here we are grabbing the pod name.
key := timeSeries.GetMetric().GetLabels()["opencensus_task"]
if key == "" {
// Labels we want to use don't necessarily exist when a cross series reducer has been used.
// So we can just use "mean" in the legend.
key = "mean"
}
ts[key] = timeSeries.GetPoints()
}

p, err := ts.Plot([]tsplot.PlotOption{tsplot.WithXAxisName("UTC"),
tsplot.WithXTimeTicks(time.Kitchen),
tsplot.WithFontSize(float64(12)),
tsplot.WithGrid(colornames.Darkgrey),
tsplot.WithTitle(metric)}...)
plot, err := tsplot.NewPlotFromTimeSeries(*ts)
if err != nil {
return err
log.Fatal(err)
}

if outDir == "" {
outDir, _ = os.Getwd()
}
saveFile := fmt.Sprintf("%s/%s-%s.png", outDir, service, metric)
p.Save(8*vg.Inch, 4*vg.Inch, saveFile)

return nil
saveFile := fmt.Sprintf("%s/test.png", "/tmp/")
return plot.Save(8*vg.Inch, 4*vg.Inch, saveFile)
}

func timeFormatOK(s string) bool {
Expand Down

0 comments on commit 739d6af

Please sign in to comment.