Use an APIReader so that we don't cache reads and can limit K8s RBAC

[mjlshen]

While validating OSD-13906 , the controller runs into this error:

W0126 22:36:53.621890       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:openshift-aws-vpce-operator:aws-vpce-operator" cannot list resource "secrets" in API group "" at the cluster scope
E0126 22:36:53.621924       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:openshift-aws-vpce-operator:aws-vpce-operator" cannot list resource "secrets" in API group "" at the cluster scope 

because client.Client tries to build a cache before performing any actions against the Kubernetes API. This is generally a good thing and decreases load on etcd, but since aws-vpce-operator is cluster-scoped, it tries to build a cache of secrets at the cluster scope as well. As we would like to restrict the K8s RBAC for this controller to only be able to get secrets from openshift-aws-vpce-operator at this time (https://github.com/openshift/aws-vpce-operator/blob/main/deploy/17_role.yaml), switching to a non-cached client, client.Reader, is the way to do this.

Ref:

• Non-caching client kubernetes-sigs/controller-runtime#144
• https://kubernetes.slack.com/archives/CAR30FCJZ/p1580118306075000

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mjlshen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mjlshen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@mjlshen: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[codecov-commenter]

Codecov Report

Merging #104 (2e1ce23) into main (62262f7) will decrease coverage by 0.04%.
The diff coverage is 33.33%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #104      +/-   ##
==========================================
- Coverage   44.66%   44.63%   -0.04%     
==========================================
  Files          28       28              
  Lines        1453     1454       +1     
==========================================
  Hits          649      649              
- Misses        723      724       +1     
  Partials       81       81              

Impacted Files	Coverage Δ
controllers/vpcendpoint/helpers.go	44.70% <0.00%> (ø)
controllers/vpcendpoint/vpcendpoint_controller.go	0.00% <0.00%> (ø)
pkg/secrets/secrets.go	52.63% <100.00%> (ø)

[robotmaxtron]

/lgtm