Generate CRDs & mappers for BigtableInstance #2288

justinsb · 2024-07-13T12:49:30Z

chore: generate CRDs for BigtableInstance from protos
chore: generate mappers for bigtable

cheftako · 2024-07-17T04:52:29Z

apis/bigtable/v1beta1/bigtableinstance_types.go

+	*/
+}
+
+/* NOTYET


If not yet, when?

So typically we / I have started doing one phase where we get the CRD yaml to match the existing CRD yaml. It's not a perfect match, the descriptions aren't exact, and we now mark status.observedGeneration as an int64. But there should be no field changes at this stage (we aren't using a new controller yet)

NOTYET is how I "stash" the fields that are going to be useful as we update the CRD, but we aren't yet adding support.

For this instance, we aren't introducing observedState yet, because it's an existing v1beta1 CRD. The only change we know we need right now for BigtableInstance is around the problems on autoscaling vs numNodes.

Not against having a TODO/NOTYET but I do think we should communicate when we would want to include it.
If we want other people to contribute, having sign posts which indicate if this fields is ready to be exposed (or not) will help.
It can indicate its time or that not to waste your effort yet.

Any suggestions? We at least want to wait till we have the direct controller. Maybe TODO-AFTER-DIRECT-CONTROLLER ?

Sure. I'd even be happy with something that said something like we want the direct controller to have a minimum 3 month burn in as a beta resource before we add fields the non direct controller did not support.

cheftako · 2024-07-17T04:57:24Z

...ons.k8s.io_v1_customresourcedefinition_bigtableinstances.bigtable.cnrm.cloud.google.com.yaml

@@ -70,12 +73,15 @@ spec:
                        cpuTarget:
                          description: The target CPU utilization for autoscaling.
                            Value must be between 10 and 80.
+                          format: int64


Is this being explicit or does this represent a change?

Ugh .. good call...

/hold

Actually, it's a change, but I think it's a good change. We should not be using integer in CRDs, we should be using int32 or int64. Here it's an int64 according to the proto.

(And we'll validate that it doesn't need to be quoted in our tests)

cheftako · 2024-07-17T05:11:28Z

dev/tools/proto-to-mapper/main.go

@@ -268,6 +268,7 @@ func (v *visitor) writeTypes(out io.Writer, msg protoreflect.MessageDescriptor)
 					goType = "map[string]int64"
 				} else {
 					fmt.Fprintf(out, "// TODO: map type %v %v\n", keyKind, valueKind)


Do we want to include field or goFieldName or jsonName here?

Good call - will do!

justinsb · 2024-07-17T11:18:02Z

I think I need to add mappings for the Cluster also - thanks for spotting the int64 change. It led me to check how this field is mapped in the proto.

/hold

justinsb · 2024-07-26T15:52:35Z

Actually, I don't think it matters yet that we aren't handling the clusters, that needs to be done in the controller, not the mappers.

/hold cancel

cheftako · 2024-07-26T16:47:09Z

dev/tools/controllerbuilder/README.md

@@ -0,0 +1,17 @@
+Examples:


This feels a bit off. It seems like its copied from the generate.sh file. Do we need to keep this up to date? What do I learn here that I wouldn't get from the script and if there is something maybe we should highlight it? If we want to explain to someone how to run a generate an individual resource, maybe we should package the script into individual resources to make that easy and then have a top level which runs them all?

Agreed - they should be in generate.sh, and we will DRY generate.sh as we add more types.

I feel like I keep deleting this file, it just won't go away! Deleted!

cheftako · 2024-07-26T16:52:37Z

dev/tools/controllerbuilder/scaffold/apis.go

@@ -139,7 +139,7 @@ func scaffoldDocFile(path string, cArgs *apis.APIArgs) error {
 	if err := WriteToFile(path, out.Bytes()); err != nil {
 		return err
 	}
-	color.HiGreen("New file added %s!\n", path)
+	color.HiGreen("New file added %s\n", path)


I would suggest we use %q for paths. I don't expect spaces, new lines etc in them but better to be safe.

In this case I removed the ! so that it is clickable. Agreed though, fixed

cheftako · 2024-07-26T16:58:34Z

pkg/controller/direct/bigtable/mapper.go

+}
+
+func BackupInfo_StartTime_FromProto(mapCtx *direct.MapContext, in *timestamppb.Timestamp) *string {
+	mapCtx.NotImplemented()


Do we have a test to ensure that passing a StartTime in a big table resource does not cause an error?

We have a fuzz test that checks the reachable functions, yes. This one is not reachable, because the BackupInfo message is not reachable. @jingyih is adding reachability testing at the proto level in another PR, so we may be able to not generate these unreachable messages in future (and thus we wouldn't need these methods)

I also think when we have the code to not emit functions that are already defined, we can generate default versions of these functions. We just don't have enough data yet on what they should look like ... need to do more resources!

Some more work in the controllerbuilder for oneof and optional enums.

cheftako · 2024-07-26T17:59:17Z

/lgtm
/approve

google-oss-prow · 2024-07-26T17:59:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheftako

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cheftako]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

justinsb force-pushed the generate_mappers_for_bigtableinstance branch 2 times, most recently from db8622b to eb2318e Compare July 16, 2024 12:33

justinsb changed the title ~~generate mappers for bigtableinstance~~ Generate CRDs & mappers for bigtableinstance Jul 16, 2024

justinsb changed the title ~~Generate CRDs & mappers for bigtableinstance~~ Generate CRDs & mappers for BigtableInstance Jul 16, 2024

justinsb requested a review from yuwenma July 16, 2024 23:19

cheftako reviewed Jul 17, 2024

View reviewed changes

google-oss-prow bot added the do-not-merge/hold label Jul 17, 2024

justinsb force-pushed the generate_mappers_for_bigtableinstance branch 6 times, most recently from 162d6b2 to 35cf679 Compare July 25, 2024 20:36

google-oss-prow bot removed the do-not-merge/hold label Jul 26, 2024

cheftako reviewed Jul 26, 2024

View reviewed changes

justinsb added 2 commits July 26, 2024 13:22

chore: generate CRDs for BigtableInstance from protos

49d7ade

chore: generate mappers for bigtable

8b11371

Some more work in the controllerbuilder for oneof and optional enums.

justinsb force-pushed the generate_mappers_for_bigtableinstance branch from 35cf679 to 8b11371 Compare July 26, 2024 17:22

google-oss-prow bot assigned cheftako Jul 26, 2024

google-oss-prow bot added the lgtm label Jul 26, 2024

google-oss-prow bot added the approved label Jul 26, 2024

google-oss-prow bot merged commit 378a294 into GoogleCloudPlatform:master Jul 26, 2024
14 checks passed

yuwenma added this to the 1.121 milestone Aug 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate CRDs & mappers for BigtableInstance #2288

Generate CRDs & mappers for BigtableInstance #2288

justinsb commented Jul 13, 2024

cheftako Jul 17, 2024

justinsb Jul 17, 2024

justinsb Jul 17, 2024

cheftako Jul 26, 2024

justinsb Jul 26, 2024

cheftako Jul 26, 2024

cheftako Jul 17, 2024

justinsb Jul 17, 2024

justinsb Jul 26, 2024

justinsb Jul 26, 2024

cheftako Jul 17, 2024

justinsb Jul 24, 2024

justinsb Jul 26, 2024

justinsb commented Jul 17, 2024

justinsb commented Jul 26, 2024

cheftako Jul 26, 2024

justinsb Jul 26, 2024

cheftako Jul 26, 2024

justinsb Jul 26, 2024

cheftako Jul 26, 2024

justinsb Jul 26, 2024

cheftako commented Jul 26, 2024

google-oss-prow bot commented Jul 26, 2024

Generate CRDs & mappers for BigtableInstance #2288

Generate CRDs & mappers for BigtableInstance #2288

Conversation

justinsb commented Jul 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Jul 17, 2024

justinsb commented Jul 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheftako commented Jul 26, 2024

google-oss-prow bot commented Jul 26, 2024