Regression in AWS authentication assume role configuration parsing around external_id

[transacid]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

there seems to be a regression after #17743. on some (but not all nodes) in our eks cluster vector segfaults with

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/aws/auth.rs:256:60

full backtrace below.
we do use an assume_role scenario with the elasticsearch sink. the weird part is that it happens only on 4 out of 32 nodes.

Configuration

elasticsearch:
  type: elasticsearch
  inputs:
    - XXXXXX
  api_version: v8
  auth:
    strategy: aws
    region: ${aws_region}
    assume_role: arn:aws:iam::XXXXXXXXXX:role/XXXXXXX-logshipper-role-XXXXXX
  aws:
    region: ${aws_region}
  tls:
    verify_certificate: true
    verify_hostname: true

Version

0.32.0

Debug Output

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/aws/auth.rs:256:60
stack backtrace:
   0:     0x556e6c761d34 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x556e6c761d34 - std::backtrace_rs::backtrace::trace_unsynchronized::hc579eb24ab204515
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x556e6c761d34 - std::sys_common::backtrace::_print_fmt::h7223525cfdbacda2
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x556e6c761d34 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbd7d55b7108d2ab8
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x556e6bdeb66f - core::fmt::rt::Argument::fmt::hb4f4a02b9bd9dd49
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/rt.rs:138:9
   5:     0x556e6bdeb66f - core::fmt::write::h6d54cd7c9e155ec5
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/mod.rs:1094:21
   6:     0x556e6c72c740 - std::io::Write::write_fmt::h6a453a71c692f63b
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/io/mod.rs:1713:15
   7:     0x556e6c7667df - std::sys_common::backtrace::_print::h1cbaa8b42678f928
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x556e6c7667df - std::sys_common::backtrace::print::h4ddf81241a51b337
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x556e6c766027 - std::panicking::default_hook::{{closure}}::hff91f1f484ade5cd
  10:     0x556e6c766f93 - std::panicking::default_hook::h21f14afd59f7aef9
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:288:9
  11:     0x556e6c766f93 - std::panicking::rust_panic_with_hook::h45f66047b14c555c
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:705:13
  12:     0x556e6c766a43 - std::panicking::begin_panic_handler::{{closure}}::h49d1a88ef0908eb4
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:595:13
  13:     0x556e6c7669d6 - std::sys_common::backtrace::__rust_end_short_backtrace::hccebf9e57f8cc425
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:151:18
  14:     0x556e6c7669c1 - rust_begin_unwind
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
  15:     0x556e6b72e3b2 - core::panicking::panic_fmt::h54ec9d0e3180a83d
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
  16:     0x556e6b72e582 - core::panicking::panic::h65157a6ac7f1357a
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:117:5
  17:     0x556e6cb7084a - vector::aws::auth::AwsAuthentication::credentials_provider::{{closure}}::hdbc8234b93aab872
  18:     0x556e6cde3817 - vector::sinks::elasticsearch::common::ElasticsearchCommon::parse_config::{{closure}}::h89a10a7502ac1965
  19:     0x556e6cddd92b - <vector::sinks::elasticsearch::config::ElasticsearchConfig as vector::config::sink::SinkConfig>::build::{{closure}}::hf19a7d1e8de3bdcb
  20:     0x556e6eb35212 - vector::topology::builder::Builder::build_sinks::{{closure}}::h299db8639813d26b
  21:     0x556e6eb266df - vector::topology::builder::build_pieces::{{closure}}::h68405278e5383aaf
  22:     0x556e6eafb4c8 - vector::topology::build_or_log_errors::{{closure}}::h8e1905d770815aae
  23:     0x556e6ebdb1f4 - vector::app::ApplicationConfig::from_opts::{{closure}}::h08d826dda48cca85
  24:     0x556e6ebd0183 - vector::app::Application::prepare_from_opts::h6df41f48d2692931
  25:     0x556e6eaccb8c - vector::app::Application::run::hd3f91d5eec8ecad5
  26:     0x556e6b8213f9 - vector::main::h5fe17e5b3d949488
  27:     0x556e6b820023 - std::sys_common::backtrace::__rust_begin_short_backtrace::h7e64a1f66112f288
  28:     0x556e6b821f9b - main
  29:     0x7f8f775abd0a - __libc_start_main
  30:     0x556e6b81ff2a - <unknown>
  31:                0x0 - <unknown>

Example Data

n/a

Additional Context

No response

References

• Allow external_id when assume_role is used in AWS sources/sinks #17739
• feat: add support for external_id in AWS assume role #17743

[ankitLu]

Working on it.

[jszwedko]

Working on it.

Thanks @ankitLu ! I wasn't expecting such a quick response 😅 I was just digging into it myself, but I'll let you take a look first 😄

[lg-d]

Is there a timeline for when this will be fixed? currently it's blocking deployment of Vector for some internal projects of ours. I'm getting the same when trying to use assume_role with the aws_s3 type.

[jszwedko]

Is there a timeline for when this will be fixed? currently it's blocking deployment of Vector for some internal projects of ours. I'm getting the same when trying to use assume_role with the aws_s3 type.

Thanks for the bump! I opened #18452 to fix this (cc/ @lg-d). For now, as a workaround, you should be able to run the previous minor version: v0.31.0.