Skip to content
This repository has been archived by the owner on Jul 28, 2024. It is now read-only.

when ResumeHttpResponse make envoy crash #417

Closed
zshmmm opened this issue Dec 21, 2023 · 1 comment
Closed

when ResumeHttpResponse make envoy crash #417

zshmmm opened this issue Dec 21, 2023 · 1 comment

Comments

@zshmmm
Copy link

zshmmm commented Dec 21, 2023

Describe the bug / error

i want to delay some response

func (ctx *injectDelayHttpContext) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
	requestPath, _ := proxywasm.GetHttpRequestHeader(":path")
	if strings.HasPrefix(string(requestPath), option.Option.Path) {
		if utils.NeedInject() {
			proxywasm.LogWarnf("inject %s, delay %d, contextID: %d", string(requestPath), option.Option.Delay, ctx.contextID)
			ctx.pluginCtx.delayResponses[ctx.contextID] = string(requestPath)
		}
	}
	return types.ActionContinue
}

func (ctx *injectDelayHttpContext) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action {
	if _, ok := ctx.pluginCtx.delayResponses[ctx.contextID]; ok {
		return types.ActionPause
	}
	return types.ActionContinue
}

then OnTick to ResumeHttpResponse
func (ctx *defaultPluginContext) OnTick() {
	ctxs := []uint32{}
	for ctxId, path := range ctx.delayResponses {
		proxywasm.LogWarnf("resume response: %d, path: %s", ctxId, path)
		ctxs = append(ctxs, ctxId)
		if err := proxywasm.SetEffectiveContext(ctxId); err != nil {
			proxywasm.LogCriticalf("failed to set effective context: %v", err)
			continue
		}
		if err := proxywasm.ResumeHttpResponse(); err != nil {
			proxywasm.LogCriticalf("failed to resume response: %v", err)
			continue
		}
	}
	for _, ctxId := range ctxs {
		delete(ctx.delayResponses, ctxId)
	}
}

but sometimes envoy crash

2023-12-20T10:36:24.678970Z     warning envoy wasm      wasm log grpc.wasm-plugin-inject: resume response: 263, path: /service.Greeter/SayHello
2023-12-20T10:36:24.679023Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x0
2023-12-20T10:36:24.679040Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2023-12-20T10:36:24.679043Z     critical        envoy backtrace Envoy version: a757eae4ff3ad133fc75efa953e6642436316f98/1.22.7-dev/Clean/RELEASE/BoringSSL
2023-12-20T10:36:24.679375Z     critical        envoy backtrace #0: __restore_rt [0x7f9b56256420]
2023-12-20T10:36:24.684436Z     critical        envoy backtrace #1: Envoy::Http::HeaderUtility::checkRequiredResponseHeaders() [0x564d10afa024]
2023-12-20T10:36:24.689229Z     critical        envoy backtrace #2: Envoy::Http::FilterManager::encodeHeaders() [0x564d10763cdb]
2023-12-20T10:36:24.693994Z     critical        envoy backtrace #3: Envoy::Http::ActiveStreamFilterBase::commonContinue() [0x564d1075e181]
2023-12-20T10:36:24.699272Z     critical        envoy backtrace #4: proxy_wasm::WasmBase::doAfterVmCallActions() [0x564d0f53a5eb]
2023-12-20T10:36:24.704267Z     critical        envoy backtrace #5: proxy_wasm::ContextBase::onResponseBody() [0x564d0f53d07c]
2023-12-20T10:36:24.709546Z     critical        envoy backtrace #6: Envoy::Extensions::Common::Wasm::Context::encodeData() [0x564d0f3390c9]
2023-12-20T10:36:24.714713Z     critical        envoy backtrace #7: Envoy::Http::FilterManager::encodeData() [0x564d1076443b]
2023-12-20T10:36:24.720848Z     critical        envoy backtrace #8: Envoy::Http::ActiveStreamFilterBase::commonContinue() [0x564d1075e1cf]
2023-12-20T10:36:24.720883Z     critical        envoy backtrace #9: proxy_wasm::WasmBase::doAfterVmCallActions() [0x564d0f53a5eb]
2023-12-20T10:36:24.720886Z     critical        envoy backtrace #10: proxy_wasm::ContextBase::onResponseBody() [0x564d0f53d07c]
2023-12-20T10:36:24.720887Z     critical        envoy backtrace #11: Envoy::Extensions::Common::Wasm::Context::encodeData() [0x564d0f3390c9]
2023-12-20T10:36:24.720889Z     critical        envoy backtrace #12: Envoy::Http::FilterManager::encodeData() [0x564d1076443b]
2023-12-20T10:36:24.720890Z     critical        envoy backtrace #13: Envoy::Http::ActiveStreamFilterBase::commonContinue() [0x564d1075e1cf]
2023-12-20T10:36:24.720892Z     critical        envoy backtrace #14: proxy_wasm::WasmBase::doAfterVmCallActions() [0x564d0f53a5eb]
2023-12-20T10:36:24.726362Z     critical        envoy backtrace #15: proxy_wasm::ContextBase::onTick() [0x564d0f53c094]
2023-12-20T10:36:24.731573Z     critical        envoy backtrace #16: Envoy::Extensions::Common::Wasm::Wasm::tickHandler() [0x564d0f34dca8]
2023-12-20T10:36:24.736766Z     critical        envoy backtrace #17: std::__1::__function::__func<>::operator()() [0x564d0f355932]
2023-12-20T10:36:24.741316Z     critical        envoy backtrace #18: event_process_active_single_queue [0x564d10b799a0]
2023-12-20T10:36:24.747789Z     critical        envoy backtrace #19: event_base_loop [0x564d10b78691]
2023-12-20T10:36:24.752784Z     critical        envoy backtrace #20: Envoy::Server::WorkerImpl::threadRoutine() [0x564d1035b9cd]
2023-12-20T10:36:24.758742Z     critical        envoy backtrace #21: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x564d10d75673]
2023-12-20T10:36:24.758796Z     critical        envoy backtrace #22: start_thread [0x7f9b5624a609]
ActiveStream 0x564d14a31600, stream_id_: 3983421899179809609&filter_manager_: 
  FilterManager 0x564d14a31678, state_.has_1xx_headers_: 0
  filter_manager_callbacks_.requestHeaders(): 
    ':method', 'POST'
    ':scheme', 'http'
    ':path', '/service.Greeter/SayHello'
    ':authority', 'server-svc.grpc.svc.cluster.local:8080'
    'content-type', 'application/grpc'
    'user-agent', 'grpc-go/1.45.0'
    'te', 'trailers'
    'grpc-timeout', '2000m'
    'x-forwarded-proto', 'http'
    'x-request-id', '21e1f994-02d1-48c5-baff-80b374bc0943'
    'x-envoy-attempt-count', '1'
    'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/grpc/sa/default;Hash=bb1ce34d8e5f08c780e71db6b7057a01b7371e9ca92b34a71064e77710079659;Subject="";URI=spiffe://cluster.local/ns/grpc/sa/default'
    'x-b3-traceid', '5d235ea4f72a4faedf4026506c1faf2c'
    'x-b3-spanid', '96d3618b7fbd4f87'
    'x-b3-parentspanid', 'df4026506c1faf2c'
    'x-b3-sampled', '0'
  filter_manager_callbacks_.requestTrailers():   null
  filter_manager_callbacks_.responseHeaders(): 
    ':status', '200'
    'content-type', 'application/grpc'
    'x-envoy-upstream-service-time', '0'
    'x-envoy-peer-metadata', 'ChoKDkFQUF9DT05UQUlORVJTEggaBnNlcnZlcgoYCgpDTFVTVEVSX0lEEgoaCGNsdXN0ZXIxChwKDElOU1RBTkNFX0lQUxIMGgoxMC40Mi4yLjkxChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xNC42CsYBCgZMQUJFTFMSuwEquAEKDwoDYXBwEggaBnNlcnZlcgohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo3YmJjZDc2YmJmCiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KKwofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIIGgZzZXJ2ZXIKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChIKB01FU0hfSUQSBxoFbWVzaDEKIQoETkFNRRIZGhdzZXJ2ZXItN2JiY2Q3NmJiZi1uamdsdAoTCglOQU1FU1BBQ0USBhoEZ3JwYwpHCgVPV05FUhI+GjxrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvZ3JwYy9kZXBsb3ltZW50cy9zZXJ2ZXIKFwoRUExBVEZPUk1fTUVUQURBVEESAioAChkKDVdPUktMT0FEX05BTUUSCBoGc2VydmVy'
    'x-envoy-peer-metadata-id', 'sidecar~10.42.2.91~server-7bbcd76bbf-njglt.grpc~grpc.svc.cluster.local'
    'date', 'Wed, 20 Dec 2023 10:36:24 GMT'
    'server', 'istio-envoy'
  filter_manager_callbacks_.responseTrailers(): 
    'grpc-status', '0'
    'grpc-message', ''
  &stream_info_: 
    StreamInfoImpl 0x564d14a31790, protocol_: 2, response_code_: 200, response_code_details_: via_upstream, attempt_count_: 1, health_check_request_: 0, route_name_: default    upstream_info_: 
      UpstreamInfoImpl 0x564d146f3218, upstream_connection_id_: 7
    OverridableRemoteConnectionInfoSetterStreamInfo 0x564d14a31790, remoteAddress(): 10.42.1.17:51506, directRemoteAddress(): 10.42.1.17:51506, localAddress(): 10.42.2.91:8080
ConnectionImpl 0x564d151d1080, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x564d14d77600, transport_protocol_: tls
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x564d14680428, remote_address_: 10.42.1.17:51506, direct_remote_address_: 10.42.1.17:51506, local_address_: 10.42.2.91:8080, server_name_: outbound_.8080_._.server-svc.grpc.svc.cluster.local
2023-12-20T10:36:24.765365Z     info    ads     ADS: "@" server-7bbcd76bbf-njglt.grpc-2 terminated
2023-12-20T10:36:24.765507Z     info    ads     ADS: "@" server-7bbcd76bbf-njglt.grpc-1 terminated
2023-12-20T10:36:24.765766Z     error   Epoch 0 exited with error: signal: segmentation fault
2023-12-20T10:36:24.765811Z     info    No more active epochs, terminating

What is your Envoy/Istio version?

a757eae4ff3ad133fc75efa953e6642436316f98/1.22.7

What is the SDK version?

v0.22.0

What is your TinyGo version?

tinygo version 0.30.0 linux/amd64 (using go version go1.20.11 and LLVM version 16.0.1)

URL or snippet of your code including Envoy configuration

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: wasm-plugin-inject
  namespace: grpc
spec:
  imagePullPolicy: Always
  imagePullSecret: cr
  phase: AUTHZ
  priority: 1000
  selector:
    matchLabels:
      app: server
  pluginConfig:
    type: delay
    path: /service.Greeter
    delay: 200
    present: 50
    # code: 502
  url: oci://xx/wasm-plugin-inject:v1.1

Additional context (Optional)

@mathetake
Copy link
Member

I am sorry, but this should not be an issue of this SDK, but rather the Envoy's implementation issue. So I would recommend you raise an issue there.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants