Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a generic version of JQ accessor, called JQGetTypedFromAccessor to parse GitHub payload #801

Merged
merged 1 commit into from
Aug 31, 2023

Conversation

jhrozek
Copy link
Contributor

@jhrozek jhrozek commented Aug 30, 2023

Addresses a comment brought up during an earlier review that the way we
parse the github payload is unsafe - in case the JQ accessor wouldn't
return anything, the value returned would have been nil (the default
value of any) which would panic on type assertion. Similarly, if the
assertion failed due to github changing its API.

Because these attributes are all required by the layer calling
JQGetTypedFromAccessor, I opted to return an error in case the
accessor doesn't find a value which would be propagated to the caller
and handled appropriately.

Other parts of the mediator where nil can be handled more gracefully
(e.g. because it is just passed to reflect.DeepEqual) still use the
old function JQGetTypedFromAccessor which returns any

Fixes: #785

@JAORMX JAORMX self-requested a review August 30, 2023 13:07
Copy link
Contributor

@JAORMX JAORMX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 30, 2023

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

I was going back and forth on this as I was creating the patch. If JQGetValuesFromAccessor doesn't find the path, it currently returns nil, nil, so nil interface and no error. And this makes code that is only ever interested in equality simpler:

func (jqe *Evaluator) Eval(ctx context.Context, pol map[string]any, obj any) error {
	for idx := range jqe.assertions {
		a := jqe.assertions[idx]
		policyVal, err := util.JQGetValuesFromAccessor(ctx, a.Policy.Def, pol)
		if err != nil {
			return fmt.Errorf("cannot get values from policy accessor: %w", err)
		}

		dataVal, err := util.JQGetValuesFromAccessor(ctx, a.Ingested.Def, obj)
		if err != nil {
			return fmt.Errorf("cannot get values from data accessor: %w", err)
		}

		// Deep compare
		if !reflect.DeepEqual(policyVal, dataVal) {
			msg := fmt.Sprintf("data does not match policy: for assertion %d, got %v, want %v",
				idx, dataVal, policyVal)

			marshalledAssertion, err := json.MarshalIndent(a, "", "  ")
			if err == nil {
				msg = fmt.Sprintf("%s\nassertion: %s", msg, string(marshalledAssertion))
			}

			return evalerrors.NewErrEvaluationFailed(msg)
		}
	}

	return nil
}

On the other hand, when parsing the JSON payload from GitHub, we typically care that the property is there and want to handle the absence as an error which is what the new generic function does:

func JQGetTypedFromAccessor[T any](ctx context.Context, path string, obj any) (T, error) {
	var out T
	var ok bool

	outAny, err := JQGetValuesFromAccessor(ctx, path, obj)
	if err != nil {
		return out, err
	}

	if outAny == nil {
		return out, NewErrNoValueFound("no value found for path %s", path)
	}

so I thought the two have different use-cases.

@evankanderson
Copy link
Member

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

I was going back and forth on this as I was creating the patch. If JQGetValuesFromAccessor doesn't find the path, it currently returns nil, nil, so nil interface and no error. And this makes code that is only ever interested in equality simpler:

I'd be okay with having to read JQGetValuesFromAccessor[any](...) in that example. If we wanted to make it more readable, we could even call it JQReadFrom[any]. 😁

evankanderson
evankanderson previously approved these changes Aug 30, 2023
Copy link
Member

@evankanderson evankanderson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love the unit tests, thank you!

internal/util/jq.go Outdated Show resolved Hide resolved
internal/util/jq.go Outdated Show resolved Hide resolved
@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 30, 2023

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

I was going back and forth on this as I was creating the patch. If JQGetValuesFromAccessor doesn't find the path, it currently returns nil, nil, so nil interface and no error. And this makes code that is only ever interested in equality simpler:

I'd be okay with having to read JQGetValuesFromAccessor[any](...) in that example. If we wanted to make it more readable, we could even call it JQReadFrom[any]. 😁

To make sure I understand you, are you saying that you'd like both functions to use generics, but let one return the default value for the type (so, nil for the case of any) and the other explicitly return an error in case either of the functions didn't find the path?

@evankanderson
Copy link
Member

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

I was going back and forth on this as I was creating the patch. If JQGetValuesFromAccessor doesn't find the path, it currently returns nil, nil, so nil interface and no error. And this makes code that is only ever interested in equality simpler:

I'd be okay with having to read JQGetValuesFromAccessor[any](...) in that example. If we wanted to make it more readable, we could even call it JQReadFrom[any]. 😁

To make sure I understand you, are you saying that you'd like both functions to use generics, but let one return the default value for the type (so, nil for the case of any) and the other explicitly return an error in case either of the functions didn't find the path?

No, I'm suggesting just one function that returns either a found value and no error, or the default value and an error if the value isn't found or is of the wrong type.

@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 30, 2023

Can we make JQGetValuesFromAccessor typed entirely? So we'd only have one function for this. This looks super useful.

I was going back and forth on this as I was creating the patch. If JQGetValuesFromAccessor doesn't find the path, it currently returns nil, nil, so nil interface and no error. And this makes code that is only ever interested in equality simpler:

I'd be okay with having to read JQGetValuesFromAccessor[any](...) in that example. If we wanted to make it more readable, we could even call it JQReadFrom[any]. 😁

To make sure I understand you, are you saying that you'd like both functions to use generics, but let one return the default value for the type (so, nil for the case of any) and the other explicitly return an error in case either of the functions didn't find the path?

No, I'm suggesting just one function that returns either a found value and no error, or the default value and an error if the value isn't found or is of the wrong type.

That's what I started with and found it awkward (or less compact at least) in the "internal/util/jq.go module. I'll submit that version though again (I must still have it somewhere in git reflog so it's no extra work) and let you and Ozz check it out.

@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 30, 2023

That's what I started with and found it awkward (or less compact at least) in the "internal/util/jq.go module. I'll submit that version though again (I must still have it somewhere in git reflog so it's no extra work) and let you and Ozz check it out.

Turns out that relying on the default value makes the code quite readable. WDYT?

evankanderson
evankanderson previously approved these changes Aug 30, 2023
Copy link
Member

@evankanderson evankanderson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks gorgeous, thank you!

…o parse GitHub payload

Addresses a comment brought up during an earlier review that the way we
parse the github payload is unsafe - in case the JQ accessor wouldn't
return anything, the value returned would have been `nil` (the default
value of `any`) which would panic on type assertion. Similarly, if the
assertion failed due to github changing its API.

Because these attributes are all required by the layer calling
`JQGetTypedFromAccessor`, I opted to return an error in case the
accessor doesn't find a value which would be propagated to the caller
and handled appropriately.

Other parts of the mediator where `nil` can be handled more gracefully
(e.g. because it is just passed to `reflect.DeepEqual`) still use the
old function `JQGetTypedFromAccessor` which returns `any`

Fixes: #785
@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 31, 2023

The latest push just fixes a lint issue (one testcase didn't call t.Parallel)

@jhrozek jhrozek merged commit dcc597c into mindersec:main Aug 31, 2023
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Type-safe artifact attribute casting when parsing JSON input
3 participants