Remove Catalogmetadata CRD and serve CatalogMetadata with aggregated apiserver

[anik120]

Motivation:

The child resources created by the Catalog controller are registered as CRDs, which are served by the main kube apiserver. These resources should instead be served by an aggregated apiservice, so that the consumption and delivery of these API objects can be customized according to catalogd's use cases (eg custom storage for the objects so that the main etcd storage is not being misused, custom CRUD permissions for the objects like "only the catalog controller can CRUD the objects, etc).

Additional background: https://hackmd.io/-Rrvhgb3Q7m9HYZW2gxQtQ

Goal:

Configure the apiserver to administer the child resources, and get rid of the CRDs (expect for the catalogsource CRD, since that doesn't pose any resource constraint risk for the main apiserver)

[joelanford]

For reference, my prototype that implements this is located at https://github.com/joelanford/catalogd

I definitely think we should copy/paste code from there and/or use it as inspiration/reference to jumpstart this work.

[anik120]

@joelanford for the PR linked above, I decided to take a slightly different route for now while I was studying your prototype. Looks like the controller and the server is the same process, and the data is also stored in the same process. ie the data is stored and accessed within the same process.

I have a slightly modified proposal:

☝🏽 that arch decouples the processor of the request (for adding a new catalog to the cluster) from the storage, and the processor and the storage from the server of the data. This'll allow us to isolate problems and build out solutions for them separately. For eg the PR uses an etcd store, but the etcd store can be switched out for a bolt db easily without affecting any other part of the solution. Or for eg, server has a single replica serving requests in a dedicated cluster, but multiple replicas are deployed for serving requests in a hosted cluster.

Building the entire solution in a single process leaves very little wiggle room for customizing solutions for different environments, so my proposal is to start off with controller+etcd storage + server, and then iterate over each part of that arch.

[anik120]

Moving this out of the milestone since we want to investigate taking a different route than an apiserver

[everettraven]

Closing this since we decided to go the route of an HTTP server for serving catalog content. See #113 , #114, and #139 for more information