Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quic: add QUIC downstream connection close error stats. #16584

Merged
merged 18 commits into from
Jun 3, 2021
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions source/common/quic/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,19 @@ envoy_cc_library(
],
)

envoy_cc_library(
name = "quic_stats_lib",
srcs = ["quic_stats.cc"],
hdrs = ["quic_stats.h"],
tags = ["nofips"],
deps = [
"//include/envoy/stats:stats_interface",
"//source/common/stats:symbol_table_lib",
"@com_googlesource_quiche//:quic_core_error_codes_lib",
"@com_googlesource_quiche//:quic_core_types_lib",
],
)

envoy_cc_library(
name = "envoy_quic_proof_source_base_lib",
srcs = ["envoy_quic_proof_source_base.cc"],
Expand Down Expand Up @@ -314,6 +327,7 @@ envoy_cc_library(
hdrs = ["envoy_quic_dispatcher.h"],
tags = ["nofips"],
deps = [
"quic_stats_lib",
":envoy_quic_proof_source_lib",
":envoy_quic_server_connection_lib",
":envoy_quic_server_session_lib",
Expand Down
17 changes: 10 additions & 7 deletions source/common/quic/active_quic_listener.cc
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,19 @@ ActiveQuicListener::ActiveQuicListener(
uint32_t worker_index, uint32_t concurrency, Event::Dispatcher& dispatcher,
Network::UdpConnectionHandler& parent, Network::ListenerConfig& listener_config,
const quic::QuicConfig& quic_config, Network::Socket::OptionsSharedPtr options,
bool kernel_worker_routing, const envoy::config::core::v3::RuntimeFeatureFlag& enabled)
bool kernel_worker_routing, const envoy::config::core::v3::RuntimeFeatureFlag& enabled,
QuicStats* quic_stats)
: ActiveQuicListener(worker_index, concurrency, dispatcher, parent,
listener_config.listenSocketFactory().getListenSocket(), listener_config,
quic_config, std::move(options), kernel_worker_routing, enabled) {}
quic_config, std::move(options), kernel_worker_routing, enabled,
quic_stats) {}

ActiveQuicListener::ActiveQuicListener(
uint32_t worker_index, uint32_t concurrency, Event::Dispatcher& dispatcher,
Network::UdpConnectionHandler& parent, Network::SocketSharedPtr listen_socket,
Network::ListenerConfig& listener_config, const quic::QuicConfig& quic_config,
Network::Socket::OptionsSharedPtr options, bool kernel_worker_routing,
const envoy::config::core::v3::RuntimeFeatureFlag& enabled)
const envoy::config::core::v3::RuntimeFeatureFlag& enabled, QuicStats* quic_stats)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest bundling QuicStats and quic::QuicConfig into a class Quic::Context, which can parallel Http::Context in most ways.

That will cut down by one on the arg-count in a few interfaces in this PR, and give you other good places to put stuff that may need to be shared across workers.

Moreover, Quic::Context (or maybe OptRef<Quic::Context>) could then be returned from Server::FactoryContext, also parallel to Http::Context and Grpc::Context, cutting down on how much plumbing through needs to happen for those individually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally tried to make QuicStats behave similarly to Http::Context but found it to be quite complicated.

To have the server return QuicStats, we need a pure interface class so that Server::Instance can have a getter interface Quic::QuicStats quicStats().
However, the only public method in QuicStats, chargeQuicConnectionCloseStats(), is dependent on QuicErrorCode, which is defined in third_party library Quiche.
In my understanding, Envoy must to be able to build without quiche. And we probably shouldn't introduce external dependency in the include/ directory either.
Having this dependency makes it hard to follow existing pattern when implementing Quic stats.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On this point, to match the pattern in the other Contexts is to forward-declare QuicStats without defining an interface for it.

So you'd have

namespace quic { class QuicConfig; }
namespace Envoy {
namespace Quic {
class QuicStats;
class Context {
 public:
  virtual QuicStats& stats() PURE:
  virtual quic::QuicConfig& config() PURE;
};
...

So you don't need to expose the error-code type or other types at the interface.

: Server::ActiveUdpListenerBase(
worker_index, concurrency, parent, *listen_socket,
dispatcher.createUdpListener(
Expand Down Expand Up @@ -81,7 +83,7 @@ ActiveQuicListener::ActiveQuicListener(
quic_dispatcher_ = std::make_unique<EnvoyQuicDispatcher>(
crypto_config_.get(), quic_config, &version_manager_, std::move(connection_helper),
std::move(alarm_factory), quic::kQuicDefaultConnectionIdLength, parent, *config_, stats_,
per_worker_stats_, dispatcher, listen_socket_);
per_worker_stats_, dispatcher, listen_socket_, quic_stats);

// Create udp_packet_writer
Network::UdpPacketWriterPtr udp_packet_writer =
Expand Down Expand Up @@ -215,8 +217,9 @@ uint32_t ActiveQuicListener::destination(const Network::UdpRecvData& data) const
}

ActiveQuicListenerFactory::ActiveQuicListenerFactory(
const envoy::config::listener::v3::QuicProtocolOptions& config, uint32_t concurrency)
: concurrency_(concurrency), enabled_(config.enabled()) {
const envoy::config::listener::v3::QuicProtocolOptions& config, uint32_t concurrency,
QuicStats* quic_stats)
: concurrency_(concurrency), enabled_(config.enabled()), quic_stats_(quic_stats) {
uint64_t idle_network_timeout_ms =
config.has_idle_timeout() ? DurationUtil::durationToMilliseconds(config.idle_timeout())
: 300000;
Expand Down Expand Up @@ -301,7 +304,7 @@ Network::ConnectionHandler::ActiveUdpListenerPtr ActiveQuicListenerFactory::crea

return std::make_unique<ActiveQuicListener>(worker_index, concurrency_, disptacher, parent,
config, quic_config_, std::move(options),
kernel_worker_routing, enabled_);
kernel_worker_routing, enabled_, quic_stats_);
} // namespace Quic

} // namespace Quic
Expand Down
9 changes: 6 additions & 3 deletions source/common/quic/active_quic_listener.h
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@ class ActiveQuicListener : public Envoy::Server::ActiveUdpListenerBase,
Network::UdpConnectionHandler& parent,
Network::ListenerConfig& listener_config, const quic::QuicConfig& quic_config,
Network::Socket::OptionsSharedPtr options, bool kernel_worker_routing,
const envoy::config::core::v3::RuntimeFeatureFlag& enabled);
const envoy::config::core::v3::RuntimeFeatureFlag& enabled,
QuicStats* quic_stats);

ActiveQuicListener(uint32_t worker_index, uint32_t concurrency, Event::Dispatcher& dispatcher,
Network::UdpConnectionHandler& parent, Network::SocketSharedPtr listen_socket,
Network::ListenerConfig& listener_config, const quic::QuicConfig& quic_config,
Network::Socket::OptionsSharedPtr options, bool kernel_worker_routing,
const envoy::config::core::v3::RuntimeFeatureFlag& enabled);
const envoy::config::core::v3::RuntimeFeatureFlag& enabled,
QuicStats* quic_stats);

~ActiveQuicListener() override;

Expand Down Expand Up @@ -82,7 +84,7 @@ class ActiveQuicListenerFactory : public Network::ActiveUdpListenerFactory,
Logger::Loggable<Logger::Id::quic> {
public:
ActiveQuicListenerFactory(const envoy::config::listener::v3::QuicProtocolOptions& config,
uint32_t concurrency);
uint32_t concurrency, QuicStats* quic_stats);

// Network::ActiveUdpListenerFactory.
Network::ConnectionHandler::ActiveUdpListenerPtr
Expand All @@ -97,6 +99,7 @@ class ActiveQuicListenerFactory : public Network::ActiveUdpListenerFactory,
const uint32_t concurrency_;
absl::once_flag install_bpf_once_;
envoy::config::core::v3::RuntimeFeatureFlag enabled_;
QuicStats* quic_stats_;
};

} // namespace Quic
Expand Down
8 changes: 6 additions & 2 deletions source/common/quic/envoy_quic_dispatcher.cc
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ EnvoyQuicDispatcher::EnvoyQuicDispatcher(
uint8_t expected_server_connection_id_length, Network::ConnectionHandler& connection_handler,
Network::ListenerConfig& listener_config, Server::ListenerStats& listener_stats,
Server::PerHandlerListenerStats& per_worker_stats, Event::Dispatcher& dispatcher,
Network::Socket& listen_socket)
Network::Socket& listen_socket, QuicStats* quic_stats)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per above, pass through QuicContext rather than QuicStats and QuicConfig separately.

: quic::QuicDispatcher(&quic_config, crypto_config, version_manager, std::move(helper),
std::make_unique<EnvoyQuicCryptoServerStreamHelper>(),
std::move(alarm_factory), expected_server_connection_id_length),
connection_handler_(connection_handler), listener_config_(listener_config),
listener_stats_(listener_stats), per_worker_stats_(per_worker_stats), dispatcher_(dispatcher),
listen_socket_(listen_socket) {
listen_socket_(listen_socket), quic_stats_(quic_stats) {
// Set send buffer twice of max flow control window to ensure that stream send
// buffer always takes all the data.
// The max amount of data buffered is the per-stream high watermark + the max
Expand All @@ -46,6 +46,10 @@ void EnvoyQuicDispatcher::OnConnectionClosed(quic::QuicConnectionId connection_i
listener_stats_.downstream_cx_active_.dec();
per_worker_stats_.downstream_cx_active_.dec();
connection_handler_.decNumConnections();
if (quic_stats_) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should just make QuicStats unconditional at this level. I don't see the problem with instantiating QuicStats in the places below where you are currently passing nullptr.

quic_stats_->chargeQuicConnectionCloseStats(listener_config_.listenerScope(), error, source,
/*is_upstream*/ false);
}
}

std::unique_ptr<quic::QuicSession> EnvoyQuicDispatcher::CreateQuicSession(
Expand Down
22 changes: 11 additions & 11 deletions source/common/quic/envoy_quic_dispatcher.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include "envoy/network/listener.h"
#include "server/connection_handler_impl.h"
#include "server/active_listener_base.h"
#include "common/quic/quic_stats.h"

namespace Envoy {
namespace Quic {
Expand All @@ -43,17 +44,15 @@ class EnvoyQuicCryptoServerStreamHelper : public quic::QuicCryptoServerStreamBas

class EnvoyQuicDispatcher : public quic::QuicDispatcher {
public:
EnvoyQuicDispatcher(const quic::QuicCryptoServerConfig* crypto_config,
const quic::QuicConfig& quic_config,
quic::QuicVersionManager* version_manager,
std::unique_ptr<quic::QuicConnectionHelperInterface> helper,
std::unique_ptr<quic::QuicAlarmFactory> alarm_factory,
uint8_t expected_server_connection_id_length,
Network::ConnectionHandler& connection_handler,
Network::ListenerConfig& listener_config,
Server::ListenerStats& listener_stats,
Server::PerHandlerListenerStats& per_worker_stats,
Event::Dispatcher& dispatcher, Network::Socket& listen_socket);
EnvoyQuicDispatcher(
const quic::QuicCryptoServerConfig* crypto_config, const quic::QuicConfig& quic_config,
quic::QuicVersionManager* version_manager,
std::unique_ptr<quic::QuicConnectionHelperInterface> helper,
std::unique_ptr<quic::QuicAlarmFactory> alarm_factory,
uint8_t expected_server_connection_id_length, Network::ConnectionHandler& connection_handler,
Network::ListenerConfig& listener_config, Server::ListenerStats& listener_stats,
Server::PerHandlerListenerStats& per_worker_stats, Event::Dispatcher& dispatcher,
Network::Socket& listen_socket, QuicStats* quic_stats);

void OnConnectionClosed(quic::QuicConnectionId connection_id, quic::QuicErrorCode error,
const std::string& error_details,
Expand Down Expand Up @@ -81,6 +80,7 @@ class EnvoyQuicDispatcher : public quic::QuicDispatcher {
Server::PerHandlerListenerStats& per_worker_stats_;
Event::Dispatcher& dispatcher_;
Network::Socket& listen_socket_;
QuicStats* quic_stats_;
};

} // namespace Quic
Expand Down
41 changes: 41 additions & 0 deletions source/common/quic/quic_stats.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#include "common/quic/quic_stats.h"

namespace Envoy {
namespace Quic {

QuicStats::QuicStats(Stats::SymbolTable& symbol_table)
: stat_name_pool_(symbol_table), symbol_table_(symbol_table),
downstream_(stat_name_pool_.add("downstream")), upstream_(stat_name_pool_.add("upstream")),
from_self_(stat_name_pool_.add("self")), from_peer_(stat_name_pool_.add("peer")) {
// Preallocate most used counters
connectionCloseStatName(quic::QUIC_NETWORK_IDLE_TIMEOUT);
}

void QuicStats::incCounter(Stats::Scope& scope, const Stats::StatNameVec& names) {
Stats::SymbolTable::StoragePtr stat_name_storage = symbol_table_.join(names);
scope.counterFromStatName(Stats::StatName(stat_name_storage.get())).inc();
}

void QuicStats::chargeQuicConnectionCloseStats(Stats::Scope& scope, quic::QuicErrorCode error_code,
quic::ConnectionCloseSource source,
bool is_upstream) {
ASSERT(&symbol_table_ == &scope.symbolTable());

const Stats::StatName connection_close = connectionCloseStatName(error_code);
incCounter(scope, {is_upstream ? upstream_ : downstream_,
source == quic::ConnectionCloseSource::FROM_SELF ? from_self_ : from_peer_,
connection_close});
}

Stats::StatName QuicStats::connectionCloseStatName(quic::QuicErrorCode error_code) {
ASSERT(error_code < quic::QUIC_LAST_ERROR);

return Stats::StatName(
connection_error_stat_names_.get(error_code, [this, error_code]() -> const uint8_t* {
return stat_name_pool_.addReturningStorage(
absl::StrCat("quic_connection_close_error_code_", QuicErrorCodeToString(error_code)));
}));
}

} // namespace Quic
} // namespace Envoy
38 changes: 38 additions & 0 deletions source/common/quic/quic_stats.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#pragma once

#include "envoy/stats/scope.h"

#include "common/common/thread.h"
#include "common/stats/symbol_table_impl.h"

#include "quiche/quic/core/quic_error_codes.h"
#include "quiche/quic/core/quic_types.h"

namespace Envoy {
namespace Quic {

class QuicStats {
public:
explicit QuicStats(Stats::SymbolTable& symbol_table);

void chargeQuicConnectionCloseStats(Stats::Scope& scope, quic::QuicErrorCode error_code,
quic::ConnectionCloseSource source, bool is_upstream);

private:
void incCounter(Stats::Scope& scope, const Stats::StatNameVec& names);

Stats::StatName connectionCloseStatName(quic::QuicErrorCode error_code);

Stats::StatNamePool stat_name_pool_;
Stats::SymbolTable& symbol_table_;
const Stats::StatName downstream_;
const Stats::StatName upstream_;
const Stats::StatName from_self_;
const Stats::StatName from_peer_;
Thread::AtomicPtrArray<const uint8_t, quic::QUIC_LAST_ERROR,
Thread::AtomicPtrAllocMode::DoNotDelete>
connection_error_stat_names_;
};

} // namespace Quic
} // namespace Envoy
5 changes: 4 additions & 1 deletion source/server/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ envoy_cc_library(
"//source/common/quic:quic_factory_lib",
"//source/common/quic:quic_transport_socket_factory_lib",
"//source/common/quic:udp_gso_batch_writer_lib",
"//source/common/quic:quic_stats_lib",
]),
)

Expand Down Expand Up @@ -531,7 +532,9 @@ envoy_cc_library(
"@envoy_api//envoy/admin/v3:pkg_cc_proto",
"@envoy_api//envoy/config/bootstrap/v2:pkg_cc_proto",
"@envoy_api//envoy/config/bootstrap/v3:pkg_cc_proto",
],
] + envoy_select_enable_http3([
"//source/common/quic:quic_stats_lib",
]),
)

envoy_cc_library(
Expand Down
2 changes: 1 addition & 1 deletion source/server/config_validation/server.cc
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ void ValidationInstance::initialize(const Options& options,
messageValidationContext().staticValidationVisitor(), *api_, options_);
Configuration::InitialImpl initial_config(bootstrap, options, *this);
admin_ = std::make_unique<Server::ValidationAdmin>(initial_config.admin().address());
listener_manager_ = std::make_unique<ListenerManagerImpl>(*this, *this, *this, false);
listener_manager_ = std::make_unique<ListenerManagerImpl>(*this, *this, *this, false, nullptr);
thread_local_.registerThread(*dispatcher_, true);
runtime_singleton_ = std::make_unique<Runtime::ScopedLoaderSingleton>(
component_factory.createRuntime(*this, initial_config));
Expand Down
44 changes: 24 additions & 20 deletions source/server/listener_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ Init::Manager& ListenerFactoryContextBaseImpl::initManager() { NOT_IMPLEMENTED_G
ListenerImpl::ListenerImpl(const envoy::config::listener::v3::Listener& config,
const std::string& version_info, ListenerManagerImpl& parent,
const std::string& name, bool added_via_api, bool workers_started,
uint64_t hash, uint32_t concurrency)
uint64_t hash, uint32_t concurrency, Quic::QuicStats* quic_stats)
: parent_(parent), address_(Network::Address::resolveProtoAddress(config.address())),
bind_to_port_(shouldBindToPort(config)),
hand_off_restored_destination_connections_(
Expand Down Expand Up @@ -289,15 +289,17 @@ ListenerImpl::ListenerImpl(const envoy::config::listener::v3::Listener& config,
open_connections_(std::make_shared<BasicResourceLimitImpl>(
std::numeric_limits<uint64_t>::max(), listener_factory_context_->runtime(),
cx_limit_runtime_key_)),
local_init_watcher_(fmt::format("Listener-local-init-watcher {}", name), [this] {
if (workers_started_) {
parent_.onListenerWarmed(*this);
} else {
// Notify Server that this listener is
// ready.
listener_init_target_.ready();
}
}) {
local_init_watcher_(fmt::format("Listener-local-init-watcher {}", name),
[this] {
if (workers_started_) {
parent_.onListenerWarmed(*this);
} else {
// Notify Server that this listener is
// ready.
listener_init_target_.ready();
}
}),
quic_stats_(quic_stats) {

const absl::optional<std::string> runtime_val =
listener_factory_context_->runtime().snapshot().get(cx_limit_runtime_key_);
Expand Down Expand Up @@ -334,7 +336,7 @@ ListenerImpl::ListenerImpl(ListenerImpl& origin,
const envoy::config::listener::v3::Listener& config,
const std::string& version_info, ListenerManagerImpl& parent,
const std::string& name, bool added_via_api, bool workers_started,
uint64_t hash, uint32_t concurrency)
uint64_t hash, uint32_t concurrency, Quic::QuicStats* quic_stats)
: parent_(parent), address_(origin.address_), bind_to_port_(shouldBindToPort(config)),
hand_off_restored_destination_connections_(
PROTOBUF_GET_WRAPPED_OR_DEFAULT(config, use_original_dst, false)),
Expand All @@ -360,10 +362,12 @@ ListenerImpl::ListenerImpl(ListenerImpl& origin,
origin.listener_factory_context_->listener_factory_context_base_, this, *this)),
filter_chain_manager_(address_, origin.listener_factory_context_->parentFactoryContext(),
initManager(), origin.filter_chain_manager_),
local_init_watcher_(fmt::format("Listener-local-init-watcher {}", name), [this] {
ASSERT(workers_started_);
parent_.inPlaceFilterChainUpdate(*this);
}) {
local_init_watcher_(fmt::format("Listener-local-init-watcher {}", name),
[this] {
ASSERT(workers_started_);
parent_.inPlaceFilterChainUpdate(*this);
}),
quic_stats_(quic_stats) {
buildAccessLog();
auto socket_type = Network::Utility::protobufAddressSocketType(config.address());
buildListenSocketOptions(socket_type);
Expand Down Expand Up @@ -403,7 +407,7 @@ void ListenerImpl::buildUdpListenerFactory(Network::Socket::Type socket_type,
if (config_.udp_listener_config().has_quic_options()) {
#if defined(ENVOY_ENABLE_QUIC)
RenjieTang marked this conversation as resolved.
Show resolved Hide resolved
udp_listener_config_->listener_factory_ = std::make_unique<Quic::ActiveQuicListenerFactory>(
config_.udp_listener_config().quic_options(), concurrency);
config_.udp_listener_config().quic_options(), concurrency, quic_stats_);
#if UDP_GSO_BATCH_WRITER_COMPILETIME_SUPPORT
// TODO(mattklein123): We should be able to use GSO without QUICHE/QUIC. Right now this causes
// non-QUIC integration tests to fail, which I haven't investigated yet. Additionally, from
Expand Down Expand Up @@ -756,10 +760,10 @@ ListenerImplPtr
ListenerImpl::newListenerWithFilterChain(const envoy::config::listener::v3::Listener& config,
bool workers_started, uint64_t hash) {
// Use WrapUnique since the constructor is private.
return absl::WrapUnique(
new ListenerImpl(*this, config, version_info_, parent_, name_, added_via_api_,
/* new new workers started state */ workers_started,
/* use new hash */ hash, parent_.server_.options().concurrency()));
return absl::WrapUnique(new ListenerImpl(
*this, config, version_info_, parent_, name_, added_via_api_,
/* new new workers started state */ workers_started,
/* use new hash */ hash, parent_.server_.options().concurrency(), quic_stats_));
}

void ListenerImpl::diffFilterChain(const ListenerImpl& another_listener,
Expand Down
Loading