Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ext_proc: send attributes #30781

Conversation

jbohanon
Copy link
Contributor

@jbohanon jbohanon commented Nov 8, 2023

extract attributes changes from #29069

Commit Message:
Introduce the ability to send attributes in the External Processing Request
Additional Description:
Risk Level:
Low
Testing:

  • Integration tests for request/response attributes

Release Notes: N/A
Platform Specific Features: N/A

Copy link

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @mattklein123
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #30781 was opened by jbohanon.

see: more, trace.

Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
@jbohanon jbohanon force-pushed the feature/ext_proc/request-response-attributes branch from d2c1b3a to 87cf9ba Compare November 8, 2023 16:50
Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
@jbohanon
Copy link
Contributor Author

jbohanon commented Nov 9, 2023

@tyxia @yanavlasov @yanjunxiang-google @htuch

Thoughts on this? It's the same work as was previously reviewed in the linked PR #29069

@@ -237,7 +308,11 @@ FilterHeadersStatus Filter::decodeHeaders(RequestHeaderMap& headers, bool end_st

FilterHeadersStatus status = FilterHeadersStatus::Continue;
if (decoding_state_.sendHeaders()) {
status = onHeaders(decoding_state_, headers, end_stream);
auto activation_ptr = Filters::Common::Expr::createActivation(decoding_state_.streamInfo(),
&headers, nullptr, nullptr);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for my own education, why do we need to use CEL parser to forward the attributes? Arent these available as part of streamInfo readily? I think @tyxia 's concern was the speed and overhead when we drag the CEL parser into the picture

Copy link
Contributor Author

@jbohanon jbohanon Nov 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Attributes, as documented here, are only implemented in CEL (as far as I can tell). I don't know why this very convenient mode of accessing this info was coupled to CEL in the first place, but because it is, CEL is used to access them here in lieu of re-implementing attributes in a more common/accessible location

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to make conditional on presence of attributes in the configuration. Otherwise the CEL context is instantiated even if no attributes were needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yanavlasov I have made the change so the CEL context is only instantiated when needed

@mattklein123
Copy link
Member

/lgtm api

Copy link
Contributor

@yanavlasov yanavlasov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/wait-any

@@ -237,7 +308,11 @@ FilterHeadersStatus Filter::decodeHeaders(RequestHeaderMap& headers, bool end_st

FilterHeadersStatus status = FilterHeadersStatus::Continue;
if (decoding_state_.sendHeaders()) {
status = onHeaders(decoding_state_, headers, end_stream);
auto activation_ptr = Filters::Common::Expr::createActivation(decoding_state_.streamInfo(),
&headers, nullptr, nullptr);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to make conditional on presence of attributes in the configuration. Otherwise the CEL context is instantiated even if no attributes were needed.

Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
@yanavlasov yanavlasov merged commit 64c6d04 into envoyproxy:main Nov 20, 2023
106 checks passed
yanjunxiang-google added a commit to yanjunxiang-google/envoy that referenced this pull request Nov 22, 2023
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
absl::flat_hash_map<std::string, Extensions::Filters::Common::Expr::ExpressionPtr> expressions;
#if defined(USE_CEL_PARSER)
for (const auto& matcher : matchers) {
auto parse_status = google::api::expr::parser::Parse(matcher);
Copy link
Member

@tyxia tyxia Nov 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CEL usage here is not correct.

We need to extend the lifetime ofparse_status here because CEL lib specifically requires that parsed expression(return of parser::Parse() ) need to outlive the compiled expression (return ofExpr::createExpression)

You can see some correct examples here and here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to that comment, the source_info must also outlive the parsed expression, but we throw it away in this common method. Is this also a bug?

Copy link
Member

@tyxia tyxia Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good observation but that is probably intentional because CHECK is not performed in Envoy:

source_info is optional for EVALUATION phase (where the return of this function is used) and is used for CHECK phase( see my comment here and working example).

So, it is fine for my CEL use case because only EVAL is performed in Envoy and both PARSE and CHECK are performed in control plane, as the split model I mentioned before. (i.e., source_info is not used)

Back to common method, I think it is fine so far and in your use case here, because CHECK has not been performed in Envoy data plane.

If we really want CHECK functionality in Envoy(probably should avoid as performance overhead), we can update that common method to use the source info in that parsed expression, i.e., return of parser::Parse(matcher) which we want to save here.
cced @kyessenov the author of common method.

yanavlasov pushed a commit that referenced this pull request Nov 28, 2023
Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>
@tyxia
Copy link
Member

tyxia commented Nov 28, 2023

@jbohanon When you add back this PR, could we re-org the changes a bit? i.e., moving these matching related functions (that were added in this PR) to the new file/class?

I think that will be cleaner and more aligned with existing structure (mutation_utils, processor_state, etc )

@yanjunxiang-google
Copy link
Contributor

Please also re-run the below commands after re-org the changes as proposed above:
bazel test --config asan-fuzzer //test/extensions/filters/http/ext_proc/unit_test_fuzz:ext_proc_unit_test_fuzz --test_timeout=300

If @jbohanon have problems to run the fuzzer test in your workspace, please let me know. I can run it for you. We can debug the fuzzer issues from there if it is still happening.

@jbohanon
Copy link
Contributor Author

I will re-org and try the fuzzer, then ping you folks on the new PR when it is up. Thanks for the attention and guidance on this!

nfuden pushed a commit to solo-io/envoy-fork that referenced this pull request Nov 30, 2023
Introduce the ability to send attributes in the External Processing Request

---------

Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
ashishb-solo pushed a commit to solo-io/envoy-fork that referenced this pull request Nov 30, 2023
Introduce the ability to send attributes in the External Processing Request

---------

Signed-off-by: Jacob Bohanon <jacob.bohanon@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants