Ignore meta resources in `fn source` #2139

Shell32-Natsu · 2021-06-01T19:53:27Z

close #2033

Meta resources will be ignored by default
--include-meta-resources flag is used to explicitly include meta resources.

frankfarzan · 2021-06-01T20:47:39Z

internal/pkg/pkg.go

+
+	fnConfigPaths, err := FunctionConfigFilePaths(pkgPath, true)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get pipeline config file paths: %w", err)


what's "pipeline config file"? Please audit the error message as usual.

e2e/testdata/fn-eval/fn-source/exclude-meta-resources-by-default/.expected/exec.sh

thirdparty/cmdconfig/commands/cmdsource/cmdsource.go

frankfarzan · 2021-06-03T22:49:05Z

e2e/testdata/fn-eval/fn-source/exclude-meta-resources-by-default-out-of-place/.expected/exec.sh

+
+set -eo pipefail
+
+# create a temporary directory


the test harness should create and delete the tmp dir automatically if bash scripts are provided. That eliminates lines 18,19,29,30. Can be a follow up.

This temp dir is only used in out-of-place test cases because it needs a directory as another place to sink. I don't sink test harness should care about this kind of details.

@droot has a PR for this #2163

We discovered a bit late that using .krmignore and special-dir (out for out-of-place), these tests could be written differently where exec.sh scripts deleting/copying files around to get the git diff to right thing. If we have that pattern working well for our testing, we can potentially switch to that style in a follow up PR. I am fine having these as-is for now.

frankfarzan · 2021-06-03T22:49:18Z

e2e/testdata/fn-eval/fn-source/exclude-meta-resources-by-default-out-of-place/.expected/exec.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail


harness should set this automatically. Can be a follow up.

I suspect that this can only be set in the script. We may need a wrapper script to do it?

frankfarzan · 2021-06-03T22:53:18Z

e2e/testdata/fn-eval/fn-source/exclude-meta-resources-by-default/.expected/exec.sh

+
+kpt fn source \
+| kpt fn eval - --image gcr.io/kpt-fn/set-namespace:v0.1.3 -- namespace=staging \
+| kpt fn sink .


you don't need the '.' , it's the default

DIR is required now as mentioned in #2032 (comment).

Create #2159 to track it.

frankfarzan · 2021-06-03T22:55:37Z

internal/pkg/pkg.go

+
+	fnConfigPaths, err := FunctionConfigFilePaths(pkgPath, true)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get config paths for the functions in pipeline: %w", err)


Same as before. "config paths" won't mean much to the user. Are you referring to values specified in configPath? It's a specific field name with a proper noun.

Updated to directly return error since wrapping it here looks not very useful.

frankfarzan · 2021-06-03T22:56:08Z

thirdparty/cmdconfig/commands/cmdsource/cmdsource.go

@@ -29,6 +33,8 @@ func GetSourceRunner(name string) *SourceRunner {
 	}
 	c.Flags().StringVar(&r.FunctionConfig, "fn-config", "",
 		"path to function config file.")
+	c.Flags().BoolVar(&r.IncludeMetaResources,
+		"include-meta-resources", false, "include package meta resources in the command output")


Why does value of flag help message differ from reference docs?

The help message is a brief description about the flag, meanwhile the reference doc includes details. This is the same for other commands.

I don't agree with this. It's weird to see to a different flag description from (--help) and reference docs. It's also a waste of time to describe the same thing twice. @mortent @droot

The current output of --help combines the reference doc and flag description. Isn't it verbose to have same sentences twice?

Actually the real bug from an end user's perspective is that flags help gets printed twice in two different forms (detailed form picked from reference/*/.md and short form from the default commandline flags here).

I don't know the real fix here, but @Shell32-Natsu can you file a bug for this, we need to address it for all the commands.

frankfarzan · 2021-06-04T18:09:34Z

lgtm. I'll let Sunil approve.

droot · 2021-06-04T21:13:20Z

e2e/testdata/fn-eval/fn-source/exclude-meta-resources-by-default-out-of-place/.expected/exec.sh

+
+set -eo pipefail
+
+# create a temporary directory


We discovered a bit late that using .krmignore and special-dir (out for out-of-place), these tests could be written differently where exec.sh scripts deleting/copying files around to get the git diff to right thing. If we have that pattern working well for our testing, we can potentially switch to that style in a follow up PR. I am fine having these as-is for now.

droot · 2021-06-04T21:16:06Z

thirdparty/cmdconfig/commands/cmdsource/cmdsource.go

@@ -29,6 +33,8 @@ func GetSourceRunner(name string) *SourceRunner {
 	}
 	c.Flags().StringVar(&r.FunctionConfig, "fn-config", "",
 		"path to function config file.")
+	c.Flags().BoolVar(&r.IncludeMetaResources,
+		"include-meta-resources", false, "include package meta resources in the command output")


Actually the real bug from an end user's perspective is that flags help gets printed twice in two different forms (detailed form picked from reference/*/.md and short form from the default commandline flags here).

I don't know the real fix here, but @Shell32-Natsu can you file a bug for this, we need to address it for all the commands.

Approved by Sunil

Shell32-Natsu · 2021-06-04T21:48:26Z

@frankfarzan Looks like now all PRs need approval from code owner?

frankfarzan · 2021-06-07T17:42:00Z

@phanimarupaka should include this in the migration doc.

Shell32-Natsu added 2 commits June 1, 2021 12:37

meta resource in fn source

7bf2a0e

add e2e tests

0366f05

Shell32-Natsu requested review from droot and frankfarzan June 1, 2021 19:53

Shell32-Natsu linked an issue Jun 1, 2021 that may be closed by this pull request

kpt fn source should handle meta resources similar to fn eval #2033

Closed

frankfarzan suggested changes Jun 1, 2021

View reviewed changes

droot requested changes Jun 2, 2021

View reviewed changes

thirdparty/cmdconfig/commands/cmdsource/cmdsource.go Show resolved Hide resolved

thirdparty/cmdconfig/commands/cmdsource/cmdsource.go Outdated Show resolved Hide resolved

code review

669febd

Shell32-Natsu requested review from droot and frankfarzan June 2, 2021 17:42

Shell32-Natsu added 2 commits June 3, 2021 10:35

Merge branch 'next' into fn-source-meta

1271500

add test cases

c2596a6

frankfarzan previously requested changes Jun 3, 2021

View reviewed changes

code review

7b85d70

Shell32-Natsu requested a review from frankfarzan June 4, 2021 02:45

droot approved these changes Jun 4, 2021

View reviewed changes

Merge branch 'next' into fn-source-meta

bf84cd5

frankfarzan approved these changes Jun 7, 2021

View reviewed changes

Shell32-Natsu merged commit c7acd74 into kptdev:next Jun 7, 2021

Shell32-Natsu deleted the fn-source-meta branch June 7, 2021 18:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore meta resources in `fn source` #2139

Ignore meta resources in `fn source` #2139

Shell32-Natsu commented Jun 1, 2021

frankfarzan Jun 1, 2021

Shell32-Natsu Jun 2, 2021

frankfarzan Jun 3, 2021

Shell32-Natsu Jun 4, 2021

Shell32-Natsu Jun 4, 2021

droot Jun 4, 2021

frankfarzan Jun 3, 2021

Shell32-Natsu Jun 4, 2021 •

edited

Loading

frankfarzan Jun 3, 2021

Shell32-Natsu Jun 4, 2021

frankfarzan Jun 3, 2021

Shell32-Natsu Jun 4, 2021

frankfarzan Jun 3, 2021

Shell32-Natsu Jun 4, 2021

frankfarzan Jun 4, 2021 •

edited

Loading

Shell32-Natsu Jun 4, 2021

droot Jun 4, 2021

Shell32-Natsu Jun 4, 2021

frankfarzan commented Jun 4, 2021

droot Jun 4, 2021

droot Jun 4, 2021

Shell32-Natsu commented Jun 4, 2021

frankfarzan commented Jun 7, 2021

Ignore meta resources in fn source #2139

Ignore meta resources in fn source #2139

Conversation

Shell32-Natsu commented Jun 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shell32-Natsu Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfarzan Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfarzan commented Jun 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shell32-Natsu commented Jun 4, 2021

frankfarzan commented Jun 7, 2021

Ignore meta resources in `fn source` #2139

Ignore meta resources in `fn source` #2139

Shell32-Natsu Jun 4, 2021 •

edited

Loading

frankfarzan Jun 4, 2021 •

edited

Loading