Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ext-authz: Envoy crashes when a gRPC service responds with UNKNOWN status. #6210

Closed
gsagula opened this issue Mar 7, 2019 · 3 comments
Closed
Assignees
Labels

Comments

@gsagula
Copy link
Member

gsagula commented Mar 7, 2019

Description:
When a gRPC authorization server responds with a message that contains google.rpc.UNKNOWN status, Envoy crashes.

Repro steps:
Implement an Authorization gRPC service that responds with a message contains a CheckResponse message with google.rpc.UNKNOWN status. Envoy should return 403 instead of crashing.

For more details emissary-ingress/emissary#1291

Call Stack:

[2019-03-06 21:27:58.435][000078][critical][assert] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] assert failure: response->status().code() != Grpc::Status::GrpcStatus::Unknown.
[2019-03-06 21:27:58.435][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0x45
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<78> obj</usr/glibc-compat/lib/libc.so.6> (If unsymbolized, use tools/stack_decode.py):
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #0 0x7fa19a53f46b (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #1 0x7fa19a540790 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<78> obj<envoy>
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #2 0x6b8b4e (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #3 0x6bae27 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #4 0xd6c0d7 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #5 0xd6b560 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #6 0xd70638 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #7 0xd7dbdb (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #8 0xd7f5aa (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #9 0xcc880c (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #10 0xe030dc (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #11 0xe04ed2 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #12 0xe04f02 (unknown)
[2019-03-06 21:27:58.436][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #13 0xe1557b (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #14 0xe024d1 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #15 0xd57508 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #16 0xd57f2b (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #17 0xa9b24a (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #18 0xa9b35e (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #19 0xa91f2c (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #20 0xa935b9 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #21 0xa9335a (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #22 0xa8fc5c (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #23 0xa952fd (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #24 0xa8890b (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #25 0xa88610 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #26 0xa8868f (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #27 0x10d6854 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #28 0x10d6f9e (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #29 0x10d9d27 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #30 0xa83f9d (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #31 0xa75b52 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #32 0xa754e5 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #33 0xa76bea (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #34 0x502bd5 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #35 0x10e8157 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #36 0x10e817c (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<78> obj</usr/glibc-compat/lib/libpthread.so.0>
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #37 0x7fa19ac10cf7 (unknown)
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<78> obj</usr/glibc-compat/lib/libc.so.6>
[2019-03-06 21:27:58.437][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<78> #38 0x7fa19a6010de (unknown)
[2019-03-06 21:27:58.438][000078][critical][backtrace] [bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 78
@gsagula
Copy link
Member Author

gsagula commented Mar 7, 2019

/assign gsagula

@stale
Copy link

stale bot commented Apr 6, 2019

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label Apr 6, 2019
@gsagula
Copy link
Member Author

gsagula commented Apr 6, 2019

Working on that.

@stale stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Apr 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants