Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add profiling to API server #743

Merged
merged 2 commits into from
Apr 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cmd/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
| DB_SSLMODE | Database SSL mode | verify-full |
| DB_SSLROOTCERT | Path to CA cert used to validate Database cert | /etc/tls/db/ca.crt |
| DB_ENABLE_AUTO_MIGRATION | Auto-migrate the database on startup (create/update schemas). For further details, refer to <https://gorm.io/docs/migration.html> | true (default) |
| PROFILING | Enable profiling server | false (default) |
| PROFILING_PORT | Profiling Server Port | 6060 (default) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi @sayan-biswas I just tried this and it looks like the PROFILING_PORT defaulting to 6060 does not take effect

{"level":"info","ts":1715275165.595052,"caller":"api/main.go:237","msg":"Profiling server listening on: "}
{"level":"info","ts":1715275165.5953324,"caller":"api/main.go:259","msg":"Prometheus server listening on: 9090"}

I tried 6060, 8008, and few others anyway, no luck. Unfortunately, you can't run command like netstat in the container.

I'll try explicitly setting it to 6060 and see what happens

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it works OK when I explicitly set PROFILING_PORT @sayan-biswas

| SERVER_PORT | gRPC and REST Server Port | 8080 (default) |
| PROMETHEUS_PORT | Prometheus Port | 9090 (default) |
| PROMETHEUS_HISTOGRAM | Enable Prometheus histogram metrics to measure latency distributions of RPCs | false (default) |
Expand Down
14 changes: 13 additions & 1 deletion cmd/api/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ import (
"go.uber.org/zap"
"go.uber.org/zap/zapcore"

_ "net/http/pprof"

"github.com/golang-jwt/jwt/v4"
grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
grpc_auth "github.com/grpc-ecosystem/go-grpc-middleware/auth"
Expand Down Expand Up @@ -150,7 +152,7 @@ func main() {

// Shared options for the logger, with a custom gRPC code to log level function.
zapOpts := []grpc_zap.Option{
grpc_zap.WithDecider(func(fullMethodName string, err error) bool {
grpc_zap.WithDecider(func(fullMethodName string, _ error) bool {
return fullMethodName != healthpb.Health_Check_FullMethodName
}),
grpc_zap.WithDurationField(func(duration time.Duration) zapcore.Field {
Expand Down Expand Up @@ -188,6 +190,16 @@ func main() {
// Allow service reflection - required for grpc_cli ls to work.
reflection.Register(gs)

// Enable profiling server
if serverConfig.PROFILING {
go func() {
log.Infof("Profiling server listening on: %s", serverConfig.PROFILING_PORT)
if err := http.ListenAndServe(":"+serverConfig.PROFILING_PORT, nil); err != nil {
log.Fatalf("Error running Profiling server: %v", err)
}
}()
}

// Set up health checks.
hs := health.NewServer()
hs.SetServingStatus("tekton.results.v1alpha2.Results", healthpb.HealthCheckResponse_SERVING)
Expand Down
4 changes: 4 additions & 0 deletions config/base/api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,7 @@ spec:
protocol: TCP
port: 9090
targetPort: 9090
- name: profiling
protocol: TCP
port: 6060
targetPort: 6060
2 changes: 2 additions & 0 deletions config/base/env/config
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ DB_NAME=tekton-results
DB_SSLMODE=disable
DB_SSLROOTCERT=
DB_ENABLE_AUTO_MIGRATION=true
PROFILING=false
PROFILING_PORT=6060
SERVER_PORT=8080
PROMETHEUS_PORT=9090
PROMETHEUS_HISTOGRAM=false
Expand Down
7 changes: 7 additions & 0 deletions docs/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -650,6 +650,13 @@ curl -k https://localhost:8080/healthz
curl -k https://localhost:8080/healthz?service=tekton.results.v1alpha2.Results
```

## Profiling

The API Server includes an HTTP server for exposing golang's debug profiles. By default, the Service is disabled and exposes debug profiles on port `:6060`. For more
details on the using the profiles, see
<https://pkg.go.dev/net/http/pprof>.


## References

- [OpenAPI Specification](openapi.yaml)
3 changes: 3 additions & 0 deletions pkg/api/server/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ type Config struct {
LOGS_BUFFER_SIZE int `mapstructure:"LOGS_BUFFER_SIZE"`
LOGS_PATH string `mapstructure:"LOGS_PATH"`

PROFILING bool `mapstructure:"PROFILING"`
PROFILING_PORT string `mapstructure:"PROFILING_PORT"`

GCS_BUCKET_NAME string `mapstructure:"GCS_BUCKET_NAME"`
STORAGE_EMULATOR_HOST string `mapstructure:"STORAGE_EMULATOR_HOST"`

Expand Down
2 changes: 1 addition & 1 deletion pkg/api/server/v1alpha2/log/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ func initConfig(ctx context.Context, cfg *server.Config) (*s3.Client, error) {
var awsConfig aws.Config
var err error
if len(cfg.S3_ENDPOINT) > 0 {
customResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...any) (aws.Endpoint, error) {
customResolver := aws.EndpointResolverWithOptionsFunc(func(_, region string, _ ...any) (aws.Endpoint, error) {
if region == cfg.S3_REGION {
return aws.Endpoint{
URL: cfg.S3_ENDPOINT,
Expand Down
2 changes: 1 addition & 1 deletion pkg/cli/client/client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func TestToken(t *testing.T) {
clientset := fake.NewSimpleClientset()
// Token is a subresource of ServiceAccount, so this
// operation looks like a SA creation w.r.t. fake clients.
clientset.PrependReactor("create", "serviceaccounts", func(action ktest.Action) (handled bool, ret runtime.Object, err error) {
clientset.PrependReactor("create", "serviceaccounts", func(_ ktest.Action) (handled bool, ret runtime.Object, err error) {
return true, &v1.TokenRequest{
Status: v1.TokenRequestStatus{
Token: "a",
Expand Down
4 changes: 2 additions & 2 deletions pkg/cli/cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ func Root() *cobra.Command {
Use: "tkn-results",
Short: "tkn CLI plugin for Tekton Results API",
Long: help,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
var overrideAPIAdr string

// Prepare to port-forward if addr config is not set
Expand Down Expand Up @@ -71,7 +71,7 @@ func Root() *cobra.Command {

return nil
},
PersistentPostRun: func(cmd *cobra.Command, args []string) {
PersistentPostRun: func(_ *cobra.Command, _ []string) {
if portForwardCloseChan != nil {
close(portForwardCloseChan)
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/internal/protoutil/protoutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ func AnyBytes(t testing.TB, m proto.Message) []byte {
// ClearOutputOnly clears any proto fields marked as OUTPUT_ONLY.
func ClearOutputOnly(pb proto.Message) {
m := pb.ProtoReflect()
m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
m.Range(func(fd protoreflect.FieldDescriptor, _ protoreflect.Value) bool {
opts := fd.Options().(*descriptorpb.FieldOptions)
for _, b := range proto.GetExtension(opts, fbpb.E_FieldBehavior).([]fbpb.FieldBehavior) {
if b == fbpb.FieldBehavior_OUTPUT_ONLY {
Expand Down
2 changes: 1 addition & 1 deletion pkg/watcher/reconciler/dynamic/dynamic.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ func NewDynamicReconciler(rc pb.ResultsClient, lc pb.LogsClient, oc ObjectClient
objectClient: oc,
cfg: cfg,
// Always true predicate.
IsReadyForDeletionFunc: func(ctx context.Context, object results.Object) (bool, error) {
IsReadyForDeletionFunc: func(_ context.Context, _ results.Object) (bool, error) {
return true, nil
},
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/watcher/reconciler/dynamic/dynamic_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ func TestReconcile_TaskRun(t *testing.T) {
// Pretend that the IsReadyForDeletion function returns an
// error.
errSomethingBad := errors.New("Something really bad happened")
r.IsReadyForDeletionFunc = func(_ context.Context, object watcherresults.Object) (bool, error) {
r.IsReadyForDeletionFunc = func(_ context.Context, _ watcherresults.Object) (bool, error) {
return false, errSomethingBad
}

Expand Down