Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(controller): add stack dump for better debugging #2360

Closed
wants to merge 1 commit into from
Closed

feat(controller): add stack dump for better debugging #2360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cmd/controlplane/api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@
kubescheme "k8s.io/client-go/kubernetes/scheme"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/record"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
ctrlcfg "sigs.k8s.io/controller-runtime/pkg/config"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"

kargoapi "github.com/akuity/kargo/api/v1alpha1"
Expand Down Expand Up @@ -168,6 +170,9 @@
},
},
},
Controller: ctrlcfg.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 175 in cmd/controlplane/api.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/api.go#L173-L175 

Added lines #L173 - L175 were not covered by tests
})
if err != nil {
return nil, nil, nil, fmt.Errorf("error initializing Kargo API manager: %w", err)
Expand Down
8 changes: 8 additions & 0 deletions cmd/controlplane/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,11 @@
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/config"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"

Expand Down Expand Up @@ -194,6 +196,9 @@
BindAddress: "0",
},
Cache: cacheOpts,
Controller: config.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 201 in cmd/controlplane/controller.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/controller.go#L199-L201 

Added lines #L199 - L201 were not covered by tests
},
)
return mgr, stagesReconcilerCfg, err
Expand Down Expand Up @@ -262,6 +267,9 @@
BindAddress: "0",
},
Cache: cacheOpts,
Controller: config.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 272 in cmd/controlplane/controller.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/controller.go#L270-L272 

Added lines #L270 - L272 were not covered by tests
},
)
}
Expand Down
5 changes: 5 additions & 0 deletions cmd/controlplane/garbage_collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
"github.com/spf13/cobra"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/config"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"

Expand Down Expand Up @@ -105,6 +107,9 @@
Metrics: server.Options{
BindAddress: "0",
},
Controller: config.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 112 in cmd/controlplane/garbage_collector.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/garbage_collector.go#L110-L112 

Added lines #L110 - L112 were not covered by tests
},
)
if err != nil {
Expand Down
5 changes: 5 additions & 0 deletions cmd/controlplane/management_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
corev1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/config"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"

Expand Down Expand Up @@ -118,6 +120,9 @@
Metrics: server.Options{
BindAddress: "0",
},
Controller: config.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 125 in cmd/controlplane/management_controller.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/management_controller.go#L123-L125 

Added lines #L123 - L125 were not covered by tests
},
)
}
5 changes: 5 additions & 0 deletions cmd/controlplane/webhooks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
corev1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/config"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"
"sigs.k8s.io/controller-runtime/pkg/webhook"

Expand Down Expand Up @@ -100,6 +102,9 @@
Metrics: server.Options{
BindAddress: "0",
},
Controller: config.Controller{
RecoverPanic: ptr.To(true),
},

Check warning on line 107 in cmd/controlplane/webhooks.go

View check run for this annotation

Codecov / codecov/patch

cmd/controlplane/webhooks.go#L105-L107 

Added lines #L105 - L107 were not covered by tests
},
)
if err != nil {
Expand Down
6 changes: 3 additions & 3 deletions internal/controller/argocd/api/v1alpha1/application_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ type OperationState struct {
}

type SyncOperationResult struct {
Revision string `json:"revision,omitempty"`
Source ApplicationSource `json:"source,omitempty"`
Sources ApplicationSources `json:"sources,omitempty"`
Revision string `json:"revision,omitempty"`
Source ApplicationSource `json:"source,omitempty"`
Sources ApplicationSources `json:"sources,omitempty"`
}
5 changes: 3 additions & 2 deletions internal/controller/promotions/promotions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import (
"context"
"fmt"
"runtime/debug"
"slices"
"strconv"
"sync"
Expand Down Expand Up @@ -344,9 +345,9 @@
defer func() {
if err := recover(); err != nil {
if theErr, ok := err.(error); ok {
logger.Error(theErr, "Promotion panic")
logger.Error(theErr, "Promotion panic", "stack", string(debug.Stack()))

Check warning on line 348 in internal/controller/promotions/promotions.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/promotions/promotions.go#L348 

Added line #L348 was not covered by tests
} else {
logger.Error(nil, "Promotion panic")
logger.Error(nil, "Promotion panic", "stack", string(debug.Stack()))
}
newStatus.Phase = kargoapi.PromotionPhaseErrored
newStatus.Message = fmt.Sprintf("%v", err)
Expand Down
Loading