Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestMetricsAreServed/with_push_error_and_FallbackConfiguration_enabled/ is flaky #6125

Open
pmalek opened this issue Jun 4, 2024 · 0 comments
Open

TestMetricsAreServed/with_push_error_and_FallbackConfiguration_enabled/ is flaky #6125

pmalek opened this issue Jun 4, 2024 · 0 comments
Labels
area/tests test-flake

Comments

@pmalek
Copy link
Member

pmalek commented Jun 4, 2024

Problem statement

TestMetricsAreServed/with_push_error_and_FallbackConfiguration_enabled seems to fail randomly on CI (and also locally):

Logs caught locally from a failing run:

    run.go:230: 2024-06-04T12:34:47+02:00       info    Diagnostics server disabled
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting controller manager     {"release": "NOT_SET", "repo": "NOT_SET", "commit": "NOT_SET"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   The ingress class name has been set     {"value": "kong"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Getting enabled options and features
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "FallbackConfiguration", "enabled": true}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "GatewayAlpha", "enabled": false}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "FillIDs", "enabled": true}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "RewriteURIs", "enabled": false}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "KongServiceFacade", "enabled": false}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Found configuration option for gated feature    {"feature": "SanitizeKonnectConfigDumps", "enabled": true}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Getting the kubernetes client configuration
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting standalone health check server
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Getting the kong admin api client configuration
    run.go:230: 2024-06-04T12:34:47+02:00       error   setup.health-check      Healthz server failed   {"error": "listen tcp :10254: bind: address already in use"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Configuring and building the controller manager
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Building the manager runtime scheme and loading apis into the scheme
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   leader election forcibly disabled
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Initializing Dataplane Client
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Emitting Kubernetes events enabled, creating an event recorder for kong-client
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting Admission Server
    run.go:230: 2024-06-04T12:34:47+02:00       info    Admission webhook server disabled
    run.go:230: 2024-06-04T12:34:47+02:00       error   Failed to resolve controller's pod to attach the apply configuration events to  {"error": "unable to get POD information (missing POD_NAME or POD_NAMESPACE environment variable"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Initializing Dataplane Synchronizer
    run.go:230: 2024-06-04T12:34:47+02:00       info    WARNING: --proxy-sync-seconds is configured for 0.100000s, in DBLESS mode this may result in problems of inconsistency in the proxy state. For DBLESS mode 3.000000s+ is recommended (3s is the default).
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Status updates disabled, skipping status updater
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Initializing Dataplane address Discovery
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting Enabled Controllers
    run.go:230: 2024-06-04T12:34:47+02:00       debug   controllers.Dynamic/GRPCRoute   All required CustomResourceDefinitions are installed, skipping DynamicCRDController set up
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Add readiness probe to health server
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting KongLicense controller
    run.go:230: 2024-06-04T12:34:47+02:00       debug   controllers.Dynamic/KongLicense All required CustomResourceDefinitions are installed, skipping DynamicCRDController set up
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Inject license getter to config translator      {"license_getter_type": "*license.KongV1Alpha1KongLicenseReconciler"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Anonymous reports disabled, skipping
    run.go:230: 2024-06-04T12:34:47+02:00       info    setup   Starting manager
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.IngressClass.netv1  Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.IngressClass.netv1  Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.EndpointSlice       Starting EventSource    {"source": "kind source: *v1.EndpointSlice"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.EndpointSlice       Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Secrets     Starting EventSource    {"source": "kind source: *v1.Secret"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Secrets     Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Service     Starting EventSource    {"source": "kind source: *v1.Service"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Service     Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.UDPIngress  Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.UDPIngress  Starting EventSource    {"source": "kind source: *v1beta1.UDPIngress"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.UDPIngress  Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Ingress.netv1       Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongIngress Starting EventSource    {"source": "kind source: *v1.KongIngress"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongIngress Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Ingress.netv1       Starting EventSource    {"source": "kind source: *v1.Ingress"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Ingress.netv1       Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.TCPIngress  Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.TCPIngress  Starting EventSource    {"source": "kind source: *v1beta1.TCPIngress"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.TCPIngress  Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.IngressClassParameters      Starting EventSource    {"source": "kind source: *v1alpha1.IngressClassParameters"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.IngressClassParameters      Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongPlugin  Starting EventSource    {"source": "kind source: *v1.KongPlugin"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongPlugin  Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongUpstreamPolicy  Starting EventSource    {"source": "kind source: *v1beta1.KongUpstreamPolicy"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongUpstreamPolicy  Starting EventSource    {"source": "kind source: *v1.Service"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongUpstreamPolicy  Starting EventSource    {"source": "kind source: *v1.HTTPRoute"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongUpstreamPolicy  Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumer        Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumer        Starting EventSource    {"source": "kind source: *v1.KongConsumer"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumer        Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumerGroup   Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumerGroup   Starting EventSource    {"source": "kind source: *v1beta1.KongConsumerGroup"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumerGroup   Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongClusterPlugin   Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongClusterPlugin   Starting EventSource    {"source": "kind source: *v1.KongClusterPlugin"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongClusterPlugin   Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.GRPCRoute   Starting EventSource    {"source": "kind source: *v1.GRPCRoute"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.GRPCRoute   Starting EventSource    {"source": "kind source: *v1.GatewayClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.GRPCRoute   Starting EventSource    {"source": "kind source: *v1.Gateway"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.GRPCRoute   Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongVault   Starting EventSource    {"source": "kind source: *v1.IngressClass"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongVault   Starting EventSource    {"source": "kind source: *v1alpha1.KongVault"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongLicense Starting EventSource    {"source": "kind source: *v1alpha1.KongLicense"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongVault   Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongLicense Starting Controller
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Fetching last good configuration from gateway clients   {"count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Fetching configuration  {"url": "http://127.0.0.1:51541"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Service     Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.IngressClassParameters      Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumer        Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Last good configuration fetched from Kong node  {"url": "http://127.0.0.1:51541"}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Parsing kubernetes objects into data-plane configuration
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Successfully built data-plane configuration
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Sending configuration to gateway clients        {"urls": ["http://127.0.0.1:51541"]}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   controllers.Service     Reconciling resource    {"CoreV1Service": {"name":"kubernetes","namespace":"default"}, "namespace": "default", "name": "kubernetes"}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Adding failure  {"resource_name": "test-service", "source": "[0]", "problem": "broken object"}
    run.go:230: 2024-06-04T12:34:47+02:00       error   recording a Warning event for object    {"name": "test-service", "namespace": "default", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationApplyFailed", "message": "invalid [0]: broken object", "error": "object failed to apply"}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   events  invalid [0]: broken object      {"type": "Warning", "object": {"kind":"Service","namespace":"default","name":"test-service","uid":"a3b8afcc-9f19-42e4-aa8f-5866168c2ad3","apiVersion":"v1"}, "reason": "KongConfigurationApplyFailed"}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Config status changed, notifying        {"configStatus": "ApplyFailed"}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   Sending configuration to gateway clients        {"urls": ["http://127.0.0.1:51541"]}
    run.go:230: 2024-06-04T12:34:47+02:00       info    Successfully synced configuration to Kong       {"url": "http://127.0.0.1:51541", "update_strategy": "InMemory"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    Successfully recovered from configuration rejection with fallback configuration
    run.go:230: 2024-06-04T12:34:47+02:00       error   dataplane-synchronizer  Could not update kong admin     {"error": "performing update for http://127.0.0.1:51541 failed: failed posting new config to /config: got status code 400"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongIngress Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Secrets     Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongUpstreamPolicy  Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongLicense Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.EndpointSlice       Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       debug   controllers.EndpointSlice       Reconciling resource    {"DiscoveryV1EndpointSlice": {"name":"kubernetes","namespace":"default"}, "namespace": "default", "name": "kubernetes"}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.Ingress.netv1       Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongClusterPlugin   Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.GRPCRoute   Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongVault   Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.UDPIngress  Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongPlugin  Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:47+02:00       info    controllers.KongConsumerGroup   Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   Parsing kubernetes objects into data-plane configuration
    run.go:230: 2024-06-04T12:34:48+02:00       debug   Successfully built data-plane configuration
    run.go:230: 2024-06-04T12:34:48+02:00       debug   Sending configuration to gateway clients        {"urls": ["http://127.0.0.1:51541"]}
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change, skipping sync to Kong  {"url": "http://127.0.0.1:51541"}
    run.go:230: 2024-06-04T12:34:48+02:00       debug   Config status changed, notifying        {"configStatus": "OK"}
    run.go:230: 2024-06-04T12:34:48+02:00       info    controllers.IngressClass.netv1  Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:48+02:00       info    controllers.TCPIngress  Starting workers        {"worker count": 1}
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change; pushing config to gateway is not necessary, skipping
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change; pushing config to gateway is not necessary, skipping
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change; pushing config to gateway is not necessary, skipping
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change; pushing config to gateway is not necessary, skipping
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
    run.go:230: 2024-06-04T12:34:48+02:00       debug   No configuration change; pushing config to gateway is not necessary, skipping
    run.go:230: 2024-06-04T12:34:48+02:00       debug   controllers.KongLicense No KongLicense available
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/tests test-flake
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pmalek @czeslavo