Package Verification Bug bufio.Scanner: token too long

[kensipe]

With the release of v0.11.0-rc1 a failure to create a package was detected resulting in a token too long

 go run cmd/kubectl-kudo/main.go package verify ../operators/repository/spark/operator/
Errors                                                             
reading from document decoder failed: bufio.Scanner: token too long
Error: found 1 package verification errors
exit status 255

It was determined that #1376 caused this break.

[porridge]

This also highlights the fact that our error messages have a lot of room for improvement - many of them lack enough context. In this case at least the path to the file being read is needed.

[nfnt]

This is YamlToObject in pkg/engine/renderer. docDecoder.Read is returning that error. Checking if this is related to #1376. The input YAML is indeed very long. Unfortunately, installing the spark operator wasn't tested as part of KUDO tests.

[nfnt]

This large input is causing the error in YamlToObject: https://github.com/kudobuilder/operators/blob/master/repository/spark/operator/templates/spark-operator-crds.yaml

[nfnt]

It's an error returned by yaml.NewDocumentDecoder. The input buffer is too large.

[nfnt]

#1376 is unrelated. We should go back to splitting input YAML ourselves, as the helpers in apimachinery don't seem to handle large input strings. cc @porridge

[nfnt]

By default, a bufio.Scanner uses MaxScanTokenSize = 64 * 1024 (https://golang.org/pkg/bufio/#pkg-constants). Our input is much larger than that.

[kensipe]

Added bug report against api-machinery ^^