Pipelines Service breaks inconsistently

[blairdrummond]

Every other invocation results in an error like this

If I had to guess, something is load-balancing over something broken, it's a fairly consistent on-off thing.

CC @sylus

[bryanpaget]

Kubeflow Pipelines is currently not working on prod. I've run the following notebook (which used to work): Kubeflow Examples. Looking at the Logs from the Kubeflow web interface, we can see errors related to reading and writing to disk (cannot save parameter ./out.txt" argo=true error="open out.txt: no such file or directory):

time="2022-07-28T13:21:20.786Z" level=info msg="capturing logs" argo=true
usage: average.py [-h] [--output_file OUTPUT_FILE] numbers [numbers ...]
average.py: error: argument numbers: invalid float value: ''
time="2022-07-28T13:21:21.474Z" level=error msg="cannot save parameter ./out.txt" argo=true error="open out.txt: no such file or directory"
time="2022-07-28T13:21:21.474Z" level=error msg="cannot save artifact ./out.txt" argo=true error="stat ./out.txt: no such file or directory"
Error: exit status 2

Therefore, we are experiencing a different issue with Kubeflow Pipelines than identified by Blair.

[bryanpaget]

Note: The above error is from prod. Running the same experiment on dev yields a different error:

ERROR:root:Failed to read a token from file '/var/run/secrets/kubeflow/pipelines/token' ([Errno 2] No such file or directory: '/var/run/secrets/kubeflow/pipelines/token').
WARNING:root:Failed to set up default credentials. Proceeding without credentials...
{'ContainerOp': {'is_exit_handler': False, 'human_name': 'average', 'display_name': None, 'name': 'average 4', 'node_selector': {}, 'volumes': [], 'tolerations': [], 'affinity': {}, 'pod_annotations': {}, 'pod_labels': {}, 'num_retries': 0, 'retry_policy': None, 'backoff_factor': None, 'backoff_duration': None, 'backoff_max_duration': None, 'timeout': 0, 'init_containers': [], 'sidecars': [], 'loop_args': None, '_component_spec_inputs_with_pipeline_params': [], '_inputs': [], 'dependent_names': [], 'enable_caching': True, 'attrs_with_pipelineparams': ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations', '_container', 'artifact_arguments', '_parameter_arguments'], '_is_v2': False, '_container': {'args': ['{{pipelineparam:op=;name=c}}',
          '{{pipelineparam:op=average 3;name=data}}',
          '{{pipelineparam:op=average 2;name=data}}'],
 'command': ['python', 'average.py'],
 'env': None,
 'env_from': None,
 'image': 'k8scc01covidacr.azurecr.io/kfp-components/average:v1',
 'image_pull_policy': None,
 'lifecycle': None,
 'liveness_probe': None,
 'ports': None,
 'readiness_probe': None,
 'resources': None,
 'security_context': None,
 'stdin': None,
 'stdin_once': None,
 'termination_message_path': None,
 'termination_message_policy': None,
 'tty': None,
 'volume_devices': None,
 'volume_mounts': None,
 'working_dir': None}, 'add_env_from': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9430>, 'add_env_variable': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9550>, 'add_port': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9670>, 'add_resource_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9790>, 'add_resource_request': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b98b0>, 'add_volume_devices': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b99d0>, 'add_volume_mount': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9af0>, 'get_resource_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9c10>, 'get_resource_request': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9d30>, 'set_cpu_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9e50>, 'set_cpu_request': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b9f70>, 'set_ephemeral_storage_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b50d0>, 'set_ephemeral_storage_request': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b51f0>, 'set_gpu_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5310>, 'set_image_pull_policy': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5430>, 'set_lifecycle': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5550>, 'set_liveness_probe': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5670>, 'set_memory_limit': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5790>, 'set_memory_request': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b58b0>, 'set_readiness_probe': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b59d0>, 'set_security_context': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5af0>, 'set_stdin': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5c10>, 'set_stdin_once': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5d30>, 'set_termination_message_path': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5e50>, 'set_termination_message_policy': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9b5f70>, 'set_tty': <function deprecation_warning.<locals>._wrapped at 0x7fc57f9bc0d0>, 'input_artifact_paths': {}, 'artifact_arguments': {}, 'file_outputs': {'data': './out.txt'}, 'output_artifact_paths': {}, '_metadata': None, '_parameter_arguments': None, 'execution_options': ExecutionOptionsSpec(retry_strategy=None, caching_strategy=CachingStrategySpec(max_cache_staleness=None)), 'outputs': {'data': {{pipelineparam:op=average 4;name=data}}}, 'output': {{pipelineparam:op=average 4;name=data}}, 'pvolumes': {}, 'pvolume': None}}
/opt/conda/lib/python3.8/site-packages/kfp/dsl/_container_op.py:1207: FutureWarning: Please create reusable components instead of constructing ContainerOp instances directly. Reusable components are shareable, portable and have compatibility and support guarantees. Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op) and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.components.html#kfp.components.load_component_from_file
  warnings.warn(
[Pipeline details](https://kubeflow.aaw-dev.cloud.statcan.ca/pipeline/#/pipelines/details/70477784-5d4a-44dd-801c-124095983ed9).
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [6], in <cell line: 5>()
      5 try:
----> 6     experiment = client.get_experiment(
      7         name=experiment_name,
      8         description=experiment_description,
      9         namespace=namespace)
     10 except:

TypeError: get_experiment() got an unexpected keyword argument 'name'

During handling of the above exception, another exception occurred:

ApiException                              Traceback (most recent call last)
Input In [6], in <cell line: 5>()
      6     experiment = client.get_experiment(
      7         name=experiment_name,
      8         description=experiment_description,
      9         namespace=namespace)
     10 except:
---> 11     experiment = client.create_experiment(
     12         name=experiment_name,
     13         description=experiment_description,
     14         namespace=namespace)
     16 run = client.run_pipeline(
     17     experiment_id=experiment.id,
     18     job_name=metadata.get("run_name"),
     19     pipeline_package_path=metadata.get("pipeline_package_path"))

File /opt/conda/lib/python3.8/site-packages/kfp/_client.py:398, in Client.create_experiment(self, name, description, namespace)
    396 experiment = None
    397 try:
--> 398   experiment = self.get_experiment(experiment_name=name, namespace=namespace)
    399 except ValueError as error:
    400   # Ignore error if the experiment does not exist.
    401   if not str(error).startswith('No experiment is found with name'):

File /opt/conda/lib/python3.8/site-packages/kfp/_client.py:510, in Client.get_experiment(self, experiment_id, experiment_name, namespace)
    500 experiment_filter = json.dumps({ 
    501     "predicates": [ 
    502       { 
   (...)
    507     ] 
    508   })
    509 if namespace:
--> 510   result = self._experiment_api.list_experiment(
    511     filter=experiment_filter,
    512     resource_reference_key_type=kfp_server_api.models.api_resource_type.ApiResourceType.NAMESPACE, 
    513     resource_reference_key_id=namespace)
    514 else:
    515   result = self._experiment_api.list_experiment(filter=experiment_filter)

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api/experiment_service_api.py:567, in ExperimentServiceApi.list_experiment(self, **kwargs)
    532 """Finds all experiments. Supports pagination, and sorting on certain fields.  # noqa: E501
    533 
    534 This method makes a synchronous HTTP request by default. To make an
   (...)
    564 :rtype: ApiListExperimentsResponse
    565 """
    566 kwargs['_return_http_data_only'] = True
--> 567 return self.list_experiment_with_http_info(**kwargs)

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api/experiment_service_api.py:668, in ExperimentServiceApi.list_experiment_with_http_info(self, **kwargs)
    665 # Authentication setting
    666 auth_settings = ['Bearer']  # noqa: E501
--> 668 return self.api_client.call_api(
    669     '/apis/v1beta1/experiments', 'GET',
    670     path_params,
    671     query_params,
    672     header_params,
    673     body=body_params,
    674     post_params=form_params,
    675     files=local_var_files,
    676     response_type='ApiListExperimentsResponse',  # noqa: E501
    677     auth_settings=auth_settings,
    678     async_req=local_var_params.get('async_req'),
    679     _return_http_data_only=local_var_params.get('_return_http_data_only'),  # noqa: E501
    680     _preload_content=local_var_params.get('_preload_content', True),
    681     _request_timeout=local_var_params.get('_request_timeout'),
    682     collection_formats=collection_formats)

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py:364, in ApiClient.call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
    327 """Makes the HTTP request (synchronous) and returns deserialized data.
    328 
    329 To make an async_req request, set the async_req parameter.
   (...)
    361     then the method will return the response directly.
    362 """
    363 if not async_req:
--> 364     return self.__call_api(resource_path, method,
    365                            path_params, query_params, header_params,
    366                            body, post_params, files,
    367                            response_type, auth_settings,
    368                            _return_http_data_only, collection_formats,
    369                            _preload_content, _request_timeout, _host)
    371 return self.pool.apply_async(self.__call_api, (resource_path,
    372                                                method, path_params,
    373                                                query_params,
   (...)
    381                                                _request_timeout,
    382                                                _host))

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py:188, in ApiClient.__call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
    186 except ApiException as e:
    187     e.body = e.body.decode('utf-8') if six.PY3 else e.body
--> 188     raise e
    190 content_type = response_data.getheader('content-type')
    192 self.last_response = response_data

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py:181, in ApiClient.__call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host)
    177     url = _host + resource_path
    179 try:
    180     # perform request and return response
--> 181     response_data = self.request(
    182         method, url, query_params=query_params, headers=header_params,
    183         post_params=post_params, body=body,
    184         _preload_content=_preload_content,
    185         _request_timeout=_request_timeout)
    186 except ApiException as e:
    187     e.body = e.body.decode('utf-8') if six.PY3 else e.body

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/api_client.py:389, in ApiClient.request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    387 """Makes the HTTP request using RESTClient."""
    388 if method == "GET":
--> 389     return self.rest_client.GET(url,
    390                                 query_params=query_params,
    391                                 _preload_content=_preload_content,
    392                                 _request_timeout=_request_timeout,
    393                                 headers=headers)
    394 elif method == "HEAD":
    395     return self.rest_client.HEAD(url,
    396                                  query_params=query_params,
    397                                  _preload_content=_preload_content,
    398                                  _request_timeout=_request_timeout,
    399                                  headers=headers)

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/rest.py:230, in RESTClientObject.GET(self, url, headers, query_params, _preload_content, _request_timeout)
    228 def GET(self, url, headers=None, query_params=None, _preload_content=True,
    229         _request_timeout=None):
--> 230     return self.request("GET", url,
    231                         headers=headers,
    232                         _preload_content=_preload_content,
    233                         _request_timeout=_request_timeout,
    234                         query_params=query_params)

File /opt/conda/lib/python3.8/site-packages/kfp_server_api/rest.py:224, in RESTClientObject.request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    221     logger.debug("response body: %s", r.data)
    223 if not 200 <= r.status <= 299:
--> 224     raise ApiException(http_resp=r)
    226 return r

ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Thu, 28 Jul 2022 14:57:17 GMT', 'x-envoy-upstream-service-time': '17', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","code":13,"message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","error_details":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngit.luolix.top/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:275\ngit.luolix.top/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngit.luolix.top/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"}]}

[bryanpaget]

These errors were reproduced by another developer, so it's not just me. Going to look into using kfp.components.load_component_from_file instead, perhaps the above errors are related to API deprication?

The API is supposed to be changing:

"FutureWarning: Please create
reusable components instead of constructing ContainerOp instances directly."

/opt/conda/lib/python3.8/site-packages/kfp/dsl/_container_op.py:1207: FutureWarning: Please create 
reusable components instead of constructing ContainerOp instances directly. Reusable components are 
shareable, portable and have compatibility and support guarantees. Please see the documentation: 
https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-
definition-file The components can be created manually (or, in case of python, using 
kfp.components.create_component_from_func or func_to_container_op) and then loaded using 
kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text:
https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.components.html#kfp.components.load_component_from_file
  warnings.warn(

[bryanpaget]

Related: StatCan/aaw-kubeflow-manifests#152 (comment)

[bryanpaget]

We're missing the volume mount and env var KF_PIPELINES_SA_TOKEN_PATH=/var/run/secrets/kubeflow/pipelines/token on dev.

[bryanpaget]

Related: #1282, for the issue on dev.

[bryanpaget]

Related: kubeflow/kubeflow#5560 (Admission webhook doesn't pick up new cert when it gets renewed by cert-manager #5560)

[rohank07]

Restarted the admission webhook deployment in kubeflow namespace to pick up the new cert.

[StanHatko]

Since Kubeflow pipelines are being removed on AAW I think this issue can be closed.