Use local definitions for k8s schema validation

[jedcunningham]

Instead of using remote schemas to validate helm chart values, we will
vendor in the definitions locally so the chart can be used without
using outbound connections.

There is a precommit hook that will ensure only the definitions we use
are included in our schema file, as there are quite a few of them all
together otherwise.

closes: #20539

[jedcunningham]

I'm not sure if it is worth the effort to only include the upstream defs we actually use, but it does cut it down from 17.5k lines to 2.6k, so figured it was worth exploring at least. Thoughts?

[potiuk]

Smart!

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

I'm not sure if it is worth the effort to only include the upstream defs we actually use, but it does cut it down from 17.5k lines to 2.6k, so figured it was worth exploring at least. Thoughts?

Makes perfect sense - especially that it's fully automated with pre-commit.

Eventually we might want to contribute (thought about it) support to add mapped loaders for refs and then we could even bundle all the definitions as separate files - https://github.com/xeipuuv/gojsonschema#loading-local-schemas - but that would have to be approved by helm maintainers and released.

[potiuk]

I'm not sure if it is worth the effort to only include the upstream defs we actually use, but it does cut it down from 17.5k lines to 2.6k, so figured it was worth exploring at least. Thoughts?

The only problem with this is a way how to use some definitions that we do not use currently. I think it is worth to do it in the way that we can also remove the vendoing easily (reversing the process). The process would loook like this;

1. if you want to work on the schema and add new stuff, run a script to un-vendor the refs.
2. make your changes
3. commit (and this will automatically vendor them in).

[jedcunningham]

The only problem with this is a way how to use some definitions that we do not use currently. I think it is worth to do it in the way that we can also remove the vendoing easily (reversing the process). The process would loook like this;

Yeah, I think an easier way would be to simply remove all io.k8s.* definitions instead of resetting to {}. That would still get us the automatic k8s defs, while leaving any custom ones in place. I just pushed that change 👍.

[potiuk]

Yeah, I think an easier way would be to simply remove all io.k8s.* definitions instead of resetting to {}. That would still get us the automatic k8s defs, while leaving any custom ones in place. I just pushed that change 👍.

Cool!