Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fault injection: add support for setting gRPC status #10841

Merged
merged 10 commits into from
Apr 27, 2020
10 changes: 9 additions & 1 deletion api/envoy/extensions/filters/http/fault/v3/fault.proto
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ option (udpa.annotations.file_status).package_version_status = ACTIVE;
// Fault Injection :ref:`configuration overview <config_http_filters_fault_injection>`.
// [#extension: envoy.filters.http.fault]

// [#next-free-field: 6]
message FaultAbort {
option (udpa.annotations.versioning).previous_message_type =
"envoy.config.filter.http.fault.v2.FaultAbort";
Expand All @@ -41,6 +42,9 @@ message FaultAbort {
// HTTP status code to use to abort the HTTP request.
uint32 http_status = 2 [(validate.rules).uint32 = {lt: 600 gte: 200}];

// gRPC status code to use to abort the gRPC request.
uint32 grpc_status = 5;

// Fault aborts are controlled via an HTTP header (if applicable).
HeaderAbort header_abort = 4;
}
Expand All @@ -50,7 +54,7 @@ message FaultAbort {
type.v3.FractionalPercent percentage = 3;
}

// [#next-free-field: 14]
// [#next-free-field: 15]
message HTTPFault {
option (udpa.annotations.versioning).previous_message_type =
"envoy.config.filter.http.fault.v2.HTTPFault";
Expand Down Expand Up @@ -133,4 +137,8 @@ message HTTPFault {
// The runtime key to override the :ref:`default <config_http_filters_fault_injection_runtime>`
// runtime. The default is: fault.http.rate_limit.response_percent
string response_rate_limit_percent_runtime = 13;

// The runtime key to override the :ref:`default <config_http_filters_fault_injection_runtime>`
// runtime. The default is: fault.http.abort.grpc_status
string abort_grpc_status_runtime = 14;
}
25 changes: 21 additions & 4 deletions docs/root/configuration/http/http_filters/fault_filter.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,17 +42,27 @@ x-envoy-fault-abort-request
In order for the header to work, :ref:`header_abort
<envoy_api_field_config.filter.http.fault.v2.FaultAbort.header_abort>` needs to be set.

x-envoy-fault-abort-grpc-request
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a % header here also? cc @Augustyniak

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think that we should add a support for x-envoy-fault-abort-grpc-request-percentage HTTP header (whose implementation would be based of how x-envoy-fault-abort-request-percentage HTTP header is defined).

Together with @buildbreaker @murki we discussed potential scenarios where having x-envoy-fault-abort-grpc-request-percentage HTTP header would be helpful to test client-side implementation of some of the GRPC requests that Lyft mobile applications perform. This lets me believe that the addition of such header will be useful for engineers from other companies too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I missed one thing when I posted my initial comment here.

Initially, I thought that GRPC aborts would be applied on top of http aborts meaning something like:

  1. Requests comes in
  2. Check whether to abort the request with http fault - look at x-envoy-fault-abort-request-percentage and x-envoy-fault-abort-request HTTP headers.
  3. If no http fault was applied (because percentage was <100% and we decided not to apply HTTP fault) check whether grpc faults should be applied - look at x-envoy-fault-abort-grpc-request-percentage and x-envoy-fault-abort-grpc-request HTTP headers.

and for this reason I thought that we needed the addition of x-envoy-fault-abort-grpc-request-percentage HTTP header.

Now that I know that the implementation looks more like apply "either HTTP or GRPC abort" (only one of them) I don't see any benefits of us having x-envoy-fault-abort-grpc-request-percentage HTTP header. Don't get me wrong, as far as I can tell current implementation works correctly and it unlocks GRPC abort tests for mobile people at Lyft but the addition of x-envoy-fault-abort-grpc-request-percentage HTTP header itself doesn't add any value in here - we could continue to use x-envoy-fault-abort-request-percentage HTTP request header and functionality-wise we wouldn't lose anything. We could keep it but I think that removal makes a little more sense since it would keep the implementation simpler.

I really feel bad asking you to do this @vijayrajput1 but could we get rid of x-envoy-fault-abort-grpc-request-percentage HTTP header and use x-envoy-fault-abort-request-percentage for both HTTP and GRPC aborts? My deep apologizes here - I know that the addition of this header costed you a significant amount of work in the first place so it really feels bad to ask you to revert some of your changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Augustyniak , I actually had this (using one header) approach in mind earlier but then I kinda convinced myself having a matching header would make things more clear. Meaning use x-envoy-fault-abort-grpc-request-percentage for x-envoy-fault-abort-grpc-request vs use x-envoy-fault-abort-request-percentage for x-envoy-fault-abort-grpc-request. I should have raised this point earlier. I'm okay, with either of those approaches - my take is if having a separate header x-envoy-fault-abort-grpc-request-percentage isn't making things anymore clear than we should use only one header i.e x-envoy-fault-abort-request-percentage - @mattklein123 any preference ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah agreed on using 1 header. Sorry, I didn't look very deeply at this before punting review over to @Augustyniak. Apologies also!

gRPC status code to abort a request with. The header value should be a non-negative integer that specifies
the gRPC status code to return in response to a request. Its value range is [0, UInt32.Max] instead of [0, 16]
to allow testing even not well-defined gRPC status codes. When this header is set, the HTTP response status code
will be set to 200. In order for the header to work, :ref:`header_abort
<envoy_api_field_config.filter.http.fault.v2.FaultAbort.header_abort>` needs to be set. If both
*x-envoy-fault-abort-request* and *x-envoy-fault-abort-grpc-request* headers are set then
*x-envoy-fault-abort-grpc-request* header will be **ignored** and fault response http status code will be
set to *x-envoy-fault-abort-request* header value.

x-envoy-fault-abort-request-percentage
The percentage of requests that should be failed with a status code that's defined
by the value of *x-envoy-fault-abort-request* HTTP header. The header value should be an integer
that specifies the numerator of the percentage of request to apply aborts to and must be greater
or equal to 0 and its maximum value is capped by the value of the numerator of
by the value of *x-envoy-fault-abort-request* or *x-envoy-fault-abort-grpc-request* HTTP headers.
The header value should be an integer that specifies the numerator of the percentage of request to apply aborts
to and must be greater or equal to 0 and its maximum value is capped by the value of the numerator of
:ref:`percentage <envoy_api_field_config.filter.http.fault.v2.FaultAbort.percentage>` field.
Percentage's denominator is equal to default percentage's denominator
:ref:`percentage <envoy_api_field_config.filter.http.fault.v2.FaultAbort.percentage>` field.
In order for the header to work, :ref:`header_abort
<envoy_api_field_config.filter.http.fault.v2.FaultAbort.header_abort>` needs to be set and
*x-envoy-fault-abort-request* HTTP header needs to be a part of a request.
either *x-envoy-fault-abort-request* or *x-envoy-fault-abort-grpc-request* HTTP header needs to be a part of the request.

x-envoy-fault-delay-request
The duration to delay a request by. The header value should be an integer that specifies the number
Expand Down Expand Up @@ -144,6 +154,13 @@ fault.http.abort.http_status
available regardless of whether the filter is :ref:`configured for abort
<envoy_api_field_config.filter.http.fault.v2.HTTPFault.abort>`.

fault.http.abort.grpc_status
gRPC status code that will be used as the response status code of requests that will be
aborted if the headers match. Defaults to the gRPC status code specified in the config.
If this field is missing from both the runtime and the config, gRPC status code in the response
will be derived from *fault.http.abort.http_status* field. This runtime key is only available when
the filter is :ref:`configured for abort <envoy_api_field_config.filter.http.fault.v2.HTTPFault.abort>`.

fault.http.delay.fixed_delay_percent
% of requests that will be delayed if the headers match. Defaults to the
*delay_percent* specified in the config or 0 otherwise. This runtime key is only available when
Expand Down
2 changes: 2 additions & 0 deletions docs/root/version_history/current.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Changes
* dynamic forward proxy: added :ref:`SNI based dynamic forward proxy <config_network_filters_sni_dynamic_forward_proxy>` support.
* fault: added support for controlling the percentage of requests that abort, delay and response rate limits faults
are applied to using :ref:`HTTP headers <config_http_filters_fault_injection_http_header>` to the HTTP fault filter.
* fault: added support for specifying grpc_status code in abort faults using
:ref:`HTTP header <config_http_filters_fault_injection_http_header>` or abort fault configuration in HTTP fault filter.
* filter: add `upstram_rq_time` stats to the GPRC stats filter.
Disabled by default and can be enabled via :ref:`enable_upstream_stats <envoy_v3_api_field_extensions.filters.http.grpc_stats.v3.FilterConfig.enable_upstream_stats>`.
* http: fixed a bug where the upgrade header was not cleared on responses to non-upgrade requests.
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 25 additions & 4 deletions source/extensions/filters/common/fault/fault_config.cc
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,13 @@ FaultAbortConfig::FaultAbortConfig(
switch (abort_config.error_type_case()) {
case envoy::extensions::filters::http::fault::v3::FaultAbort::ErrorTypeCase::kHttpStatus:
provider_ =
std::make_unique<FixedAbortProvider>(abort_config.http_status(), abort_config.percentage());
std::make_unique<FixedAbortProvider>(static_cast<Http::Code>(abort_config.http_status()),
absl::nullopt, abort_config.percentage());
break;
case envoy::extensions::filters::http::fault::v3::FaultAbort::ErrorTypeCase::kGrpcStatus:
provider_ = std::make_unique<FixedAbortProvider>(
absl::nullopt, static_cast<Grpc::Status::GrpcStatus>(abort_config.grpc_status()),
abort_config.percentage());
break;
case envoy::extensions::filters::http::fault::v3::FaultAbort::ErrorTypeCase::kHeaderAbort:
provider_ = std::make_unique<HeaderAbortProvider>(abort_config.percentage());
Expand All @@ -44,10 +50,10 @@ FaultAbortConfig::FaultAbortConfig(
}
}

absl::optional<Http::Code> FaultAbortConfig::HeaderAbortProvider::statusCode(
absl::optional<Http::Code> FaultAbortConfig::HeaderAbortProvider::httpStatusCode(
const Http::RequestHeaderMap* request_headers) const {
absl::optional<Http::Code> ret;
const auto header = request_headers->get(HeaderNames::get().AbortRequest);
absl::optional<Http::Code> ret = absl::nullopt;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that we need = absl::nullopt part here - I think that we should get rid of it if it's not needed.

Copy link
Contributor Author

@vijayrajput1 vijayrajput1 Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep its not required, I put that in there to make it apparent. I had to read absl::optional default details to find out this, so I thought making it apparent might help readability. I can remove it if its something basic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine either way. I wouldn't worry about it unless we need another push.

auto header = request_headers->get(Filters::Common::Fault::HeaderNames::get().AbortRequest);
if (header == nullptr) {
return ret;
}
Expand All @@ -64,6 +70,21 @@ absl::optional<Http::Code> FaultAbortConfig::HeaderAbortProvider::statusCode(
return ret;
}

absl::optional<Grpc::Status::GrpcStatus> FaultAbortConfig::HeaderAbortProvider::grpcStatusCode(
const Http::RequestHeaderMap* request_headers) const {
auto header = request_headers->get(Filters::Common::Fault::HeaderNames::get().AbortGrpcRequest);
if (header == nullptr) {
return absl::nullopt;
}

uint64_t code;
if (!absl::SimpleAtoi(header->value().getStringView(), &code)) {
return absl::nullopt;
}

return static_cast<Grpc::Status::GrpcStatus>(code);
}

FaultDelayConfig::FaultDelayConfig(
const envoy::extensions::filters::common::fault::v3::FaultDelay& delay_config) {
switch (delay_config.fault_delay_secifier_case()) {
Expand Down
76 changes: 55 additions & 21 deletions source/extensions/filters/common/fault/fault_config.h
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

#include "envoy/extensions/filters/common/fault/v3/fault.pb.h"
#include "envoy/extensions/filters/http/fault/v3/fault.pb.h"
#include "envoy/grpc/status.h"
#include "envoy/http/header_map.h"
#include "envoy/type/v3/percent.pb.h"

Expand All @@ -22,6 +23,7 @@ class HeaderNameValues {
const Http::LowerCaseString AbortRequest{absl::StrCat(prefix(), "-fault-abort-request")};
const Http::LowerCaseString AbortRequestPercentage{
absl::StrCat(prefix(), "-fault-abort-request-percentage")};
const Http::LowerCaseString AbortGrpcRequest{absl::StrCat(prefix(), "-fault-abort-grpc-request")};
const Http::LowerCaseString DelayRequest{absl::StrCat(prefix(), "-fault-delay-request")};
const Http::LowerCaseString DelayRequestPercentage{
absl::StrCat(prefix(), "-fault-delay-request-percentage")};
Expand Down Expand Up @@ -53,8 +55,12 @@ class FaultAbortConfig {
public:
FaultAbortConfig(const envoy::extensions::filters::http::fault::v3::FaultAbort& abort_config);

absl::optional<Http::Code> statusCode(const Http::RequestHeaderMap* request_headers) const {
return provider_->statusCode(request_headers);
absl::optional<Http::Code> httpStatusCode(const Http::RequestHeaderMap* request_headers) const {
return provider_->httpStatusCode(request_headers);
}
absl::optional<Grpc::Status::GrpcStatus>
grpcStatusCode(const Http::RequestHeaderMap* request_headers) const {
return provider_->grpcStatusCode(request_headers);
}

envoy::type::v3::FractionalPercent
Expand All @@ -71,46 +77,66 @@ class FaultAbortConfig {
// Return the HTTP status code to use. Optionally passed HTTP headers that may contain the
// HTTP status code depending on the provider implementation.
virtual absl::optional<Http::Code>
statusCode(const Http::RequestHeaderMap* request_headers) const PURE;
httpStatusCode(const Http::RequestHeaderMap* request_headers) const PURE;

// Return the gRPC status code to use. Optionally passed an HTTP header that may contain the
// gRPC status code depending on the provider implementation.
virtual absl::optional<Grpc::Status::GrpcStatus>
grpcStatusCode(const Http::RequestHeaderMap* request_headers) const PURE;

// Return what percentage of requests abort faults should be applied to. Optionally passed
// HTTP headers that may contain the percentage depending on the provider implementation.
virtual envoy::type::v3::FractionalPercent
percentage(const Http::RequestHeaderMap* request_headers) const PURE;
};

// Delay provider that uses a fixed abort status code.
// Abort provider that uses a fixed abort status code.
class FixedAbortProvider : public AbortProvider {
public:
FixedAbortProvider(uint64_t status_code, const envoy::type::v3::FractionalPercent& percentage)
: status_code_(status_code), percentage_(percentage) {}
FixedAbortProvider(absl::optional<Http::Code> http_status_code,
absl::optional<Grpc::Status::GrpcStatus> grpc_status_code,
const envoy::type::v3::FractionalPercent& percentage)
: http_status_code_(http_status_code), grpc_status_code_(grpc_status_code),
percentage_(percentage) {}

// AbortProvider
absl::optional<Http::Code> statusCode(const Http::RequestHeaderMap*) const override {
return static_cast<Http::Code>(status_code_);
absl::optional<Http::Code> httpStatusCode(const Http::RequestHeaderMap*) const override {
return http_status_code_;
}

absl::optional<Grpc::Status::GrpcStatus>
grpcStatusCode(const Http::RequestHeaderMap*) const override {
return grpc_status_code_;
}

envoy::type::v3::FractionalPercent percentage(const Http::RequestHeaderMap*) const override {
return percentage_;
}

private:
const uint64_t status_code_;
const absl::optional<Http::Code> http_status_code_;
const absl::optional<Grpc::Status::GrpcStatus> grpc_status_code_;
const envoy::type::v3::FractionalPercent percentage_;
};

// Abort provider the reads a status code from an HTTP header.
class HeaderAbortProvider : public AbortProvider, public HeaderPercentageProvider {
class HeaderAbortProvider : public AbortProvider {
public:
HeaderAbortProvider(const envoy::type::v3::FractionalPercent& percentage)
: HeaderPercentageProvider(HeaderNames::get().AbortRequestPercentage, percentage) {}
// AbortProvider
: header_percentage_provider_(HeaderNames::get().AbortRequestPercentage, percentage) {}

absl::optional<Http::Code>
statusCode(const Http::RequestHeaderMap* request_headers) const override;
httpStatusCode(const Http::RequestHeaderMap* request_headers) const override;

absl::optional<Grpc::Status::GrpcStatus>
grpcStatusCode(const Http::RequestHeaderMap* request_headers) const override;

envoy::type::v3::FractionalPercent
percentage(const Http::RequestHeaderMap* request_headers) const override {
return HeaderPercentageProvider::percentage(request_headers);
return header_percentage_provider_.percentage(request_headers);
}

private:
HeaderPercentageProvider header_percentage_provider_;
};

using AbortProviderPtr = std::unique_ptr<AbortProvider>;
Expand Down Expand Up @@ -176,18 +202,22 @@ class FaultDelayConfig {
};

// Delay provider the reads a delay from an HTTP header.
class HeaderDelayProvider : public DelayProvider, public HeaderPercentageProvider {
class HeaderDelayProvider : public DelayProvider {
public:
HeaderDelayProvider(const envoy::type::v3::FractionalPercent& percentage)
: HeaderPercentageProvider(HeaderNames::get().DelayRequestPercentage, percentage) {}
: header_percentage_provider_(HeaderNames::get().DelayRequestPercentage, percentage) {}

// DelayProvider
absl::optional<std::chrono::milliseconds>
duration(const Http::RequestHeaderMap* request_headers) const override;

envoy::type::v3::FractionalPercent
percentage(const Http::RequestHeaderMap* request_headers) const override {
return HeaderPercentageProvider::percentage(request_headers);
return header_percentage_provider_.percentage(request_headers);
}

private:
HeaderPercentageProvider header_percentage_provider_;
};

using DelayProviderPtr = std::unique_ptr<DelayProvider>;
Expand Down Expand Up @@ -252,16 +282,20 @@ class FaultRateLimitConfig {
};

// Rate limit provider that reads the rate limit from an HTTP header.
class HeaderRateLimitProvider : public RateLimitProvider, public HeaderPercentageProvider {
class HeaderRateLimitProvider : public RateLimitProvider {
public:
HeaderRateLimitProvider(const envoy::type::v3::FractionalPercent& percentage)
: HeaderPercentageProvider(HeaderNames::get().ThroughputResponsePercentage, percentage) {}
: header_percentage_provider_(HeaderNames::get().ThroughputResponsePercentage, percentage) {
}
// RateLimitProvider
absl::optional<uint64_t> rateKbps(const Http::RequestHeaderMap* request_headers) const override;
envoy::type::v3::FractionalPercent
percentage(const Http::RequestHeaderMap* request_headers) const override {
return HeaderPercentageProvider::percentage(request_headers);
return header_percentage_provider_.percentage(request_headers);
}

private:
HeaderPercentageProvider header_percentage_provider_;
};

using RateLimitProviderPtr = std::unique_ptr<RateLimitProvider>;
Expand Down
Loading