Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(kubernetes_logs source): Adds tests for checkpointing regressions in k8s #6564

Merged
merged 3 commits into from
Jun 14, 2021
Merged

chore(kubernetes_logs source): Adds tests for checkpointing regressions in k8s #6564

merged 3 commits into from
Jun 14, 2021

Conversation

eeyun
Copy link
Contributor

@eeyun eeyun commented Feb 25, 2021

Signed-off-by: Ian Henry ianjhenry00@gmail.com

@eeyun eeyun requested a review from a team February 25, 2021 22:24
@jszwedko jszwedko added ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR labels Apr 15, 2021
jszwedko added a commit that referenced this pull request Apr 15, 2021
@jszwedko
fix(kubernetes_logs source): Always use file checkpoints if they exist
60d5df1 
The `kubernetes_logs` source exposes a `PathProvider` that breaks one of
the `FileServer`s assumptions that all available files will be listed at
Vector startup time. Instead, the files are only returned once the k8s
metadata is available to the `kubernetes_logs` source. This caused the
`FileServer` to ignore any checkpoints that existed for these files.

As a short-term fix, we just always use the checkpoint, if available,
for any new files that are seen. This fixes the case for the
`kubernetes_logs` source where they are seen as "new" after start-up.

#6564 exists to test this
behavior, but it seems to pass even without this change, so that test
will need to be updated.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
@jszwedko jszwedko mentioned this pull request Apr 15, 2021
Merged
@jszwedko
Copy link
Member

So, looking at CI, it seems like these tests sometimes catch the checkpointing issue, but not consistently. I think they need to be updated to catch it more consistently. #7140 fixes this issue, so the fixes can be undone to test this test.

Also, these tests are great! Testing end-to-end k8s is no easy feat.

jszwedko added a commit that referenced this pull request Apr 16, 2021
@jszwedko
fix(kubernetes_logs source): Always use file checkpoints if they exist (
ff969e9 
#7140)

* fix(kubernetes_logs source): Always use file checkpoints if they exist

The `kubernetes_logs` source exposes a `PathProvider` that breaks one of
the `FileServer`s assumptions that all available files will be listed at
Vector startup time. Instead, the files are only returned once the k8s
metadata is available to the `kubernetes_logs` source. This caused the
`FileServer` to ignore any checkpoints that existed for these files.

As a short-term fix, we just always use the checkpoint, if available,
for any new files that are seen. This fixes the case for the
`kubernetes_logs` source where they are seen as "new" after start-up.

#6564 exists to test this
behavior, but it seems to pass even without this change, so that test
will need to be updated.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
@spencergilbert spencergilbert self-requested a review May 11, 2021 15:25
@StephenWakely
Copy link
Contributor

@jszwedko I can confirm this test does not capture the issues that arose from #7140. Would you be able to guide me through the steps you went through to reproduce that issue manually?

@jszwedko
Copy link
Member

@jszwedko I can confirm this test does not capture the issues that arose from #7140. Would you be able to guide me through the steps you went through to reproduce that issue manually?

Sure thing! It was roughly:

  1. Start minikube
  2. Deploy vector using our daemonset helm chart (helm install --namespace vector --create-namespace vector timberio/vector-agent --values values.yaml)
  3. Run flog as a k8s "job" to generate some fake logs (https://hub.docker.com/r/mingrammer/flog)
  4. Observe that vector has ingested the logs (via kubectl logs since the default vector config just dumps them to stdout)
  5. Restart the vector daemonset (kubectl rollout restart -n vector daemonset/vector-agent)
  6. Observe that the new vector pod has re-ingested the same logs via kubectl logs

This is from memory though. Let me know if that doesn't do it for you.

@StephenWakely
Copy link
Contributor 

5\. Restart the vector daemonset (`kubectl rollout restart -n vector daemonset/vector-agent`)

Curious. That is pretty much what the test is doing, in particular this line. I'll see if I can repro it manually..

langecode added a commit to neticdk/vector that referenced this pull request May 31, 2021
@langecode @dependabot
@jszwedko @ktff @shumin1027 @StephenWakely @saez0pub @blt @eeyun @Mercury2699 @pablosichert @leebenson @AnIrishDuck @moogstuart @prognant @jqueuniet @bruceg @Tolsto @tarfu @martinlevesque @spencergilbert @hhromic @oktal @apollo13 @web-flow @tobz @JeanMertz @binarylogic @lukesteensen @Shogobg
Feature/upstream update (#16)
cd16104 
* chore(deps): bump tokio-util from 0.6.5 to 0.6.6 (#7103)

Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.5 to 0.6.6.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.5...tokio-util-0.6.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(ci): bump actions/cache from v2.1.4 to v2.1.5 (#7104)

Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.4...1a9e2138d905efd099035b49d8b7a3888c653ca8)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(datadog service): Document region parameter (#7093)

* fix(datadog service): Document region parameter

Closes #7079

This was added to the shared Datadog documentation in
https://github.com/timberio/vector/pull/4174 but never added
specifically to the sinks.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* Add missing cue config for region

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(socket source): Remove socket file on source shutdown (#7047)

* Delete socket

Signed-off-by: ktf <krunotf@gmail.com>

* Add tests

Signed-off-by: ktf <krunotf@gmail.com>

* Remove spaces

Signed-off-by: ktf <krunotf@gmail.com>

* chore(deps): bump reqwest from 0.11.2 to 0.11.3 (#7100)

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.2 to 0.11.3.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.2...v0.11.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(clickhouse sink):  Fix clickhouse sink incorrectly encodes arrays (#7081)

* fix bug:clickhouse sink incorrectly encodes arrays

Signed-off-by: shumin <shumin1027@gmail.com>

* Add test for clickhouse arrays

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

Co-authored-by: shumin <shumin@compubiq.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* Added collect_subtypes. (#7092)

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* fix(docker_logs source): Docker logs missing when container has a tty (#7111)

* fix(docker_logs source): Docker logs missing when container has a tty

Enum bollard::container::LogOutput is Console when docker started
with a tty. LogOutput::Console was missing in the new_event of a
stream value received

https://docs.rs/bollard/0.10.1/bollard/container/enum.LogOutput.html

Fixes #5903

Signed-off-by: Jean Prat <jean.prat@sinch.com>

* chore(ci): Remove unneeded advisory from deny.toml (#7114)

This advisory seems to no longer be valid.

Closes: #6223

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(external docs): Fix link to unit testing documentation (#7122)

Fixes #7121

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): Upgrade cue to 0.3.2 (#7128)

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(external docs): Document additional encoding fields (#7127)

Ensure `except_fields`, `only_fields`, and `timestamp_format` appear on
all sinks supporting `encoding`. Previously they only appeared on sinks
that had healthchecks.

Fixes: #6949

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(ci): Add advisory RUSTSEC-2021-0013 back to  deny.toml (#7133)

* chore(ci): Add advisory RUSTSEC-2021-0013 back to  deny.toml

I neglected to enable all features. The `wasm` dependencies still depend
on an affected version of rust-cpuid.

This reverts commit f30d4c6ed43838b36769753f2a591c2378162e04.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* Run cargo deny in CI if deny.toml changes

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump async-trait from 0.1.48 to 0.1.49 (#7131)

Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.48 to 0.1.49.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.48...0.1.49)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(ci): Ensure all benchmark artifacts are included (#7113)

* fix(ci): Ensure all benchmark artaficts are included

CRITERION_HOME having "s was causing it to put it in the wrong place
(locally it created a directory called ").

Also just upload `./target/criterion` since Github does the
zipping for us. We did need to zip when previously downloading the
artifact in the same workflow as this was not zipped yet and so we were
hitting limits trying to individually pull a very large number of files.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump tokio from 1.4.0 to 1.5.0 (#7102)

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.4.0...tokio-1.5.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump notify from 4.0.15 to 4.0.16 (#7142)

Bumps [notify](https://github.com/notify-rs/notify) from 4.0.15 to 4.0.16.
- [Release notes](https://github.com/notify-rs/notify/releases)
- [Changelog](https://github.com/notify-rs/notify/blob/v4.0.16/CHANGELOG.md)
- [Commits](https://github.com/notify-rs/notify/compare/v4.0.15...v4.0.16)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(remap): Preserve type defs when assigning fields (#7118)

* fix(remap): Preserve type defs when assigning fields

Previously, remap would overwrite the type def of `.` whenever a new
field was assigned. That is:

```
.foo = 5
.bar = 6
```

Would result in the compiler only having type info for `.bar`.

This change causes it to merge the typedefs whenever a list of path
segments appears.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump pretty_assertions from 0.7.1 to 0.7.2 (#7143)

Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v0.7.1...v0.7.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(kubernetes_logs source): Always use file checkpoints if they exist (#7140)

* fix(kubernetes_logs source): Always use file checkpoints if they exist

The `kubernetes_logs` source exposes a `PathProvider` that breaks one of
the `FileServer`s assumptions that all available files will be listed at
Vector startup time. Instead, the files are only returned once the k8s
metadata is available to the `kubernetes_logs` source. This caused the
`FileServer` to ignore any checkpoints that existed for these files.

As a short-term fix, we just always use the checkpoint, if available,
for any new files that are seen. This fixes the case for the
`kubernetes_logs` source where they are seen as "new" after start-up.

https://github.com/timberio/vector/pull/6564 exists to test this
behavior, but it seems to pass even without this change, so that test
will need to be updated.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(remap): Fix case_sensitive defaults for string matching functions (#7091)

Fixes #7044

The docs and examples indicated that it should default to case
sensitive; however, `contains`, `starts_with`, and `ends_with`,
defaulted to case insensitive matching.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(compression): Switch to MultiGzDecoder instead of GzDecoder (#7138)

* fix(compression): Switch to MultiGzDecoder instead ef GzDecoder

Fixes #7061

It appears that AWS's ALB logging gzip's multi-part files which we were
only reading the first part of. I tested that `MultiGzDecoder` works on
simple gzip files so I figured we should switch to it everywhere. It was
already being used by the `file` source.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* Add test for multi-part zst files

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(performance): Fix remap benches (#7155)

* chore(performance): Fix remap benches

I broke them in https://github.com/timberio/vector/pull/7118

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): `cargo upgrade` (#7154)

I backed out a few changes:

```
Upgrading grok v~1.0.1 -> v1.2.0
Upgrading async-graphql-warp v=2.6.4 -> v2.8.2
Upgrading once_cell v1.3 -> v1.7.2
Upgrading async-graphql v=2.6.4 -> v2.8.2
```

Due to incompatibilities in package dependencies.

I backed out

```
Upgrading db-key v0.0.5 -> v0.1.0
```

Because it requires code changes.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump rust_decimal from 1.10.3 to 1.11.0 (#7160)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.10.3 to 1.11.0.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.10.3...1.11.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(docker_logs source): ensure that docker labels are flattened (#7152)

Fixes: #5716

Previously labels on docker containers were inserted in such a way that
dots in their names would end up creating a nested structure in the log
event due to the semantics of `LogEvent.insert`.

For example:

```json
{
  "container_created_at": "2021-04-16T18:53:19.946155600Z",
  "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319",
  "container_name": "docker_echo_1",
  "host": "COMP-C02DV25MML87",
  "image": "hashicorp/http-echo:latest",
  "label": {
    "com": {
      "docker": {
        "compose": {
          "config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e",
          "container-number": "1",
          "oneoff": "False",
          "project": {
            "config_files": "docker-compose.yml",
            "working_dir": "/private/tmp/docker"
          },
          "service": "echo",
          "version": "1.27.4"
        }
      }
    }
  },
  "message": "2021/04/16 19:14:10 localhost:5678 172.29.0.1:61824 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 35.6µs",
  "source_type": "docker",
  "stream": "stdout",
  "timestamp": "2021-04-16T19:14:10.400790400Z"
}
```

This change ensures that labels are inserted as-is as keys:

```json
{
  "container_created_at": "2021-04-16T18:53:19.946155600Z",
  "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319",
  "container_name": "docker_echo_1",
  "host": "COMP-C02DV25MML87",
  "image": "hashicorp/http-echo:latest",
  "label": {
    "com.docker.compose.config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e",
    "com.docker.compose.container-number": "1",
    "com.docker.compose.oneoff": "False",
    "com.docker.compose.project": "docker",
    "com.docker.compose.project.config_files": "docker-compose.yml",
    "com.docker.compose.project.working_dir": "/private/tmp/docker",
    "com.docker.compose.service": "echo",
    "com.docker.compose.version": "1.27.4"
  },
  "message": "2021/04/16 19:12:01 localhost:5678 172.29.0.1:61820 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 18.1µs",
  "source_type": "docker",
  "stream": "stdout",
  "timestamp": "2021-04-16T19:12:01.622769500Z"
}
```

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump gouth from 0.2.0 to 0.2.1 (#7162)

Bumps [gouth](https://github.com/mechiru/gouth) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/mechiru/gouth/releases)
- [Commits](https://github.com/mechiru/gouth/compare/v0.2.0...v0.2.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump async-trait from 0.1.49 to 0.1.50 (#7164)

Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.49 to 0.1.50.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.49...0.1.50)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): Upgrade curl-sys (#7174)

Resolves CVE-2021-22876 CVE-2021-22890

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore: Extract Core RFC (#7135)

This commit introduces a new RFC for extracting the core of vector. Closes #7027.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore: Create skeleton `lib/core` (#7178)

* Create skeleton `lib/core`

Per RFC 7027 we intend to create an extracted vector core. This is the first
step in that RFC's plan of action, to create a skeleton package "core" to
migrate code into. There's very little going on here, just a new library -- with
aggressive clippy settings that we might back out later -- and a correctness and
benchmark test.

Closes #7147

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Add newline to the end of Cargo.toml

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove unused benchmark

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(deps): bump rust_decimal from 1.11.0 to 1.11.1 (#7182)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.0 to 1.11.1.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.11.0...1.11.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump httpmock from 0.5.7 to 0.5.8 (#7181)

Bumps [httpmock](https://github.com/alexliesenfeld/httpmock) from 0.5.7 to 0.5.8.
- [Release notes](https://github.com/alexliesenfeld/httpmock/releases)
- [Changelog](https://github.com/alexliesenfeld/httpmock/blob/master/CHANGELOG.md)
- [Commits](https://github.com/alexliesenfeld/httpmock/compare/v0.5.7...v0.5.8)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(kubernetes_logs source): Refactor stream and invocation errors to support recoverable error types (#6816)

* Refactor stream and invocation errors to support recoverable error types

Signed-off-by: Ian Henry <ianjhenry00@gmail.com>

* feat(remap): add to_regex function (#7074)

* add to_regex function

Signed-off-by: Jake He <jhe@homex.com>
Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(host_metrics source): Document symlink warning workaround (#7179)

* chore(host_metrics source): Document symlink warning workaround

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* Fix spacing

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* Add adjustments to the kubernetes helm chart to match documentation

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* enhancement(remap): Support identifiers with leading numeric characters as fields in path (#7045)

* Support integers in path

Signed-off-by: ktf <krunotf@gmail.com>

* fmt

Signed-off-by: ktf <krunotf@gmail.com>

* Refactor

Signed-off-by: ktf <krunotf@gmail.com>

* New way

Signed-off-by: ktf <krunotf@gmail.com>

* Remove support for .0

Signed-off-by: ktf <krunotf@gmail.com>

* Add docs.

Signed-off-by: ktf <krunotf@gmail.com>

* Update name

Signed-off-by: ktf <krunotf@gmail.com>

* Rename `core` to `vector-core` (#7195)

Rust has its own crate `core` and I didn't anticipate that our own core would
collide with it. In retrospect it surely does.

REF #7148

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(releasing): Merge down of 0.13.0 release (#7194)

* chore(releasing): Prep 0.13.0 release

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(dev): Bump to 0.14.0 (#7205)

Now that 0.13.0 is released.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(remap): Escape quotes and backslashes when printing VRL strings (#6739)

* Escape quotes when displaying VRL strings

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Escape backslashes when displaying VRL strings

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Move shared library to dev dependencies

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Escape newlines when displaying VRL strings

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump cidr-utils from 0.5.1 to 0.5.3 (#7211)

Bumps [cidr-utils](https://github.com/magiclen/cidr-utils) from 0.5.1 to 0.5.3.
- [Release notes](https://github.com/magiclen/cidr-utils/releases)
- [Commits](https://github.com/magiclen/cidr-utils/compare/v0.5.1...v0.5.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump async-stream from 0.3.0 to 0.3.1 (#7209)

Bumps [async-stream](https://github.com/tokio-rs/async-stream) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/tokio-rs/async-stream/releases)
- [Commits](https://github.com/tokio-rs/async-stream/compare/v0.3.0...v0.3.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump syn from 1.0.69 to 1.0.70 (#7210)

Bumps [syn](https://github.com/dtolnay/syn) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.69...1.0.70)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump hyper from 0.14.5 to 0.14.6 (#7212)

Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.5 to 0.14.6.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.5...v0.14.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Move `Result` and `Error` into vector-core (#7207)

These two types are required for the `Event` interface. I have re-exported them
back in the top-level package so no code changes are necessary at that level.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove RUSTSEC-2020-0053 from ignore list since `dirs` is maintained again (#7208)

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* chore(deps): Remove RUSTSEC-2020-0146 from deny.toml (#7216)

Appears to no longer be relevant.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(performance): Consider dropping jemalloc (#6781)

I was originally going to table this after some discussion with Luke about the poor performance of musl's liballoc in multi-threaded environment, however, I had some more discussion with Brian (thank you!) that convinced me that defaulting to the system allocator may still be a better default. An argument follows:

The primary motivator for using musl, and thus greatly benefiting from jemalloc in multi-threaded environments, is with alpine docker images. Users typically choose alpine images for size reasons; however, we also publish distroless images that include libc that are only 5MB larger (which, coincidentally is also bump in size that results from including jemalloc; https://github.com/timberio/vector/pull/310).

0.12.1 images:

```
timberio/vector                               0.12.1-distroless-libc     a01c8409bacb   4 days ago      93MB
timberio/vector                               0.12.1-distroless-static   b61e0880d16c   4 days ago      82.4MB
timberio/vector                               0.12.1-alpine              ef029e56b5ee   4 days ago      88MB
timberio/vector                               0.12.1-debian              2ccea0b5ffdb   4 days ago      164MB
```

Given this, it seems like we can encourage users to install the `0.12.1-distroless-libc` image as the image of choice for size conscious installs. Distroless also has the security advantage by providing less in the container itself .

Additionally, jemalloc is historically preferred for certain workloads but it doesn't seem like we have an understanding of vector's allocation profile yet and which strategies will result in the best performance. Until we have a very strong reason to, just defaulting to the system allocator seems like a good default.

Thus the proposal is to:

* Default to the system allocator.
* Make it easy to swap the allocator to test out different allocators in the future.
* Strongly highlight this change with the 0.13 release and encourage anyone using a musl-based build to consider whether the glibc build would be usable for them.
* Document this performance difference between glibc and musl's liballoc.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(ci): bump actions/github-script from v3.1 to v4.0.1 (#7213)

Bumps [actions/github-script](https://github.com/actions/github-script) from v3.1 to v4.0.1.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v3.1...85e88a66eaa831097093a3d278536947f2984d20)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add a cargo-hack workflow (#7224)

In #7200 I ran into #7222 and figured we might do well to make cargo-hack use
official. These runs will take _a long_ time considering the feature surface of
this project so I've made it strictly a nightly run.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore: Adjust import of `crate::Event` to `crate::event::Event` (#7200)

* Adjust import of `crate::Event` to `crate::event::Event`

My end goal here is to move the `Event` and related types into
`vector-core`. This is possible but trickier if we maintain a top-level `Event`
import in vector and so here I've adjusted all imports of the relevant type to
`crate::event::Event`, removing the top-level `Event`.

This is all mechanical work and I have intentionally not changed any logic, in
deference to the size of this PR.

REF #7148

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Correct benchmark compilation failure

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* More build fixes

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* More build fixes, wasm

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* another bench import fix

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(deps): bump hyper from 0.14.6 to 0.14.7 (#7232)

Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.6 to 0.14.7.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.6...v0.14.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump regex from 1.4.5 to 1.4.6 (#7233)

Bumps [regex](https://github.com/rust-lang/regex) from 1.4.5 to 1.4.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.5...1.4.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(external docs): Correct highlight dates for 0.13.0

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump async-nats from 0.9.12 to 0.9.14 (#7234)

Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.12 to 0.9.14.
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nats-io/nats.rs/compare/0.9.12...0.9.14)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: Run cargo-hack on self-hosted machines (#7227)

* Run cargo-hack on self-hosted machines

We found in the previous run of cargo-hack that it was OOM'ed. While we were
expecting a build failure -- see #7222 -- we did not expect _that_ failure.
Run on self-hosted hardware to get more memory for the build.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Update .github/workflows/cargo_hack.yml

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* don't consume all memory

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* don't clean-per-run

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* remove PR run directive

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(buffers): Upgrade DiskBuffer to futures 0.3 (#7165)

* Update writer

Signed-off-by: ktf <krunotf@gmail.com>

* Update reader

Signed-off-by: ktf <krunotf@gmail.com>

* Update tests

Signed-off-by: ktf <krunotf@gmail.com>

* Cleanup

Signed-off-by: ktf <krunotf@gmail.com>

* Update benches

Signed-off-by: ktf <krunotf@gmail.com>

* Fix benches

Signed-off-by: ktf <krunotf@gmail.com>

* Use start_send

Signed-off-by: ktf <krunotf@gmail.com>

* Bump

Signed-off-by: ktf <krunotf@gmail.com>

* Fmt

Signed-off-by: ktf <krunotf@gmail.com>

* Add comment

Signed-off-by: ktf <krunotf@gmail.com>

* chore(splunk_hec source): Update to `futures` 0.3 (#7225)

* Update source

Signed-off-by: ktf <krunotf@gmail.com>

* Use forward

Signed-off-by: ktf <krunotf@gmail.com>

* enhancement(graphql api): Expose `events_in` & `events_out` metrics (#6888)

* Add in/out to API

Signed-off-by: ktf <krunotf@gmail.com>

* Update client

Signed-off-by: ktf <krunotf@gmail.com>

* Update tests

Signed-off-by: ktf <krunotf@gmail.com>

* Add new line

Signed-off-by: ktf <krunotf@gmail.com>

* Apply suggestions

Co-authored-by: Lee Benson <lee@leebenson.com>
Signed-off-by: ktf <krunotf@gmail.com>

* Update schema

Signed-off-by: ktf <krunotf@gmail.com>

Co-authored-by: Lee Benson <lee@leebenson.com>

* chore(deps): bump rust_decimal from 1.11.1 to 1.12.2 (#7243)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.1 to 1.12.2.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.11.1...1.12.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump tokio-postgres from 0.7.1 to 0.7.2 (#7244)

Bumps [tokio-postgres](https://github.com/sfackler/rust-postgres) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/sfackler/rust-postgres/releases)
- [Commits](https://github.com/sfackler/rust-postgres/compare/tokio-postgres-v0.7.1...tokio-postgres-v0.7.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(ci): bump actions/github-script from v4.0.1 to v4.0.2 (#7245)

Bumps [actions/github-script](https://github.com/actions/github-script) from v4.0.1 to v4.0.2.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v4.0.1...a3e7071a34d7e1f219a8a4de9a5e0a34d1ee1293)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump libc from 0.2.93 to 0.2.94 (#7254)

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.93 to 0.2.94.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.93...0.2.94)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(external docs): Document VRL |= operator (#7252)

Closes #6900

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* enhancement(kubernetes_logs source): Allow configuration of max_line_bytes (#7226)

* enhancement(kubernetes_logs source): Allow configuration of max_line_bytes

Fixes: #6966

This is specifically to allow a user to workaround the too-low hardcoded
value that was being used.

We have https://github.com/timberio/vector/issues/6967 to address that.

I lost the logs that the user provided that ran over that limit but
I believe it was due to having a very large number of labels in the
metadata for each message which is not included in the normal 16 KB
docker limit.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump tower from 0.4.6 to 0.4.7 (#7260)

Bumps [tower](https://github.com/tower-rs/tower) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/tower-rs/tower/releases)
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.6...tower-0.4.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump syn from 1.0.70 to 1.0.71 (#7261)

Bumps [syn](https://github.com/dtolnay/syn) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.70...1.0.71)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(kafka source): Fix runaway memory usage of kafka source (#7266)

* fix(kafka source): Fix runaway memory usage of kafka source

Fixes: #7258

Previously, we were cloning the `Pipeline` for each message which, due
to the behavior of futures::sync::mpsc::channel resulted in the
effective channel buffer growing by one for each message rather than any
back pressure.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(external docs): typo (#7268)

* chore(deps): bump openssl from 0.10.33 to 0.10.34 (#7270)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.33 to 0.10.34.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.33...openssl-v0.10.34)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump pulsar from 2.0.0 to 2.0.1 (#7271)

Bumps [pulsar](https://github.com/wyyerd/pulsar-rs) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/wyyerd/pulsar-rs/releases)
- [Commits](https://github.com/wyyerd/pulsar-rs/compare/v2.0.0...v2.0.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(releasing): Update install script to 0.13.1 (#7275)

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* feat(new source): Initial `exec` source (#6876)

* feat(exec source): new `exec` source (#992) (#7)

Added exec source with streaming and scheduled capability.
Generates log events from stdout and stderr of a command execution.

Signed-off-by: Stuart Broad <stuart@moogsoft.com>

* enhancement: created new lookup library (#7087)

* Created new lookup library

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Rudimentary form of lookup working with Vrl.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Lookup can parse paths itself.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Restored VRL coalesce

Signed-off-by: Stephen Wakely <stephen.wakely@datadoghq.com>

* Updated property tests.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Fixed issue with inserting a path that changes the type.

Plus a number of other tidy ups.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Clippy and formatting.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Removed unneeded file

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Removed commented out Path.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Formatting

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Vrl needs to include the . when displaying paths.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Feedback from blt.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Fixes following feedback from Jean.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Fixes following feedback from Bruce.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Clippy

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* More clippy fixes

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Renamed clone_lookup to be to_lookup

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Enable negative indexing

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Added quickcheck tests for lookups

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Fixed setting zero index

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Add limit to alternative component path

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Removed irrelevant tests

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Clippy

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Move rerun-if-changed instruction

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Tidied some comments.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Don't serialize metadata

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Update types when array index is changed.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Removed insert_null.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Clippy

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Minor comment fixes

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Remove serialize test as it is incorrect

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Feedback from Bruce.

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* Don't wipe out metadata on vrl remove

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* chore(deps): bump rust_decimal from 1.12.2 to 1.12.3 (#7280)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.2 to 1.12.3.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/compare/1.12.2...1.12.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump libz-sys from 1.1.2 to 1.1.3 (#7278)

Bumps [libz-sys](https://github.com/rust-lang/libz-sys) from 1.1.2 to 1.1.3.
- [Release notes](https://github.com/rust-lang/libz-sys/releases)
- [Commits](https://github.com/rust-lang/libz-sys/compare/1.1.2...1.1.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump async-nats from 0.9.14 to 0.9.16 (#7279)

Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.14 to 0.9.16.
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nats-io/nats.rs/compare/0.9.14...0.9.16)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: Update `top` with `events_in_total` and `events_out_total` metrics (#7286)

* Use in/out

Signed-off-by: ktf <krunotf@gmail.com>

* Remove processed_events

Signed-off-by: ktf <krunotf@gmail.com>

* Remove empty line

Signed-off-by: ktf <krunotf@gmail.com>

* Events header

Signed-off-by: ktf <krunotf@gmail.com>

* Update column width

Signed-off-by: ktf <krunotf@gmail.com>

* Pluralize

Signed-off-by: ktf <krunotf@gmail.com>

* Fixed doc tests from lookup (#7288)

Signed-off-by: Stephen Wakely <fungus.humungus@gmail.com>

* chore(ci): Fix action-discord version for test workflow (#7285)

Accidentally replaced with 0.3.2 when upgrading cue due to an eager
find/replace.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(graphql api): `--no-default-features --features api` build failure (#7284)

* host metrics

Signed-off-by: Lee Benson <lee@leebenson.com>

* -pub

Signed-off-by: Lee Benson <lee@leebenson.com>

* fix(external docs): Document scrape_interval_secs for `internal_metrics` source (#7289)

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(tests): Simplify `cargo test` usage (#7287)

We currently use `--all-targets` to include benches (`--bench`), but `--all-targets` does not include `--doc` apparently (https://github.com/rust-lang/cargo/issues/6669). Instead of using `--all-targets` we just mark benches with `test = true` so that they will be tested along with the default targets (https://doc.rust-lang.org/cargo/commands/cargo-test.html#target-selection)

Also:

* Avoid setting `SCOPE` by default in the `Makefile` as this causes examples to be skipped for some reason.
* Use `make test` for Windows to take advantage of existing make targets. I tried `make test-components` there too, but it failed to build krb5.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump tracing-core from 0.1.17 to 0.1.18 (#7294)

Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.17 to 0.1.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.17...tracing-core-0.1.18)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump tui from 0.14.0 to 0.15.0 (#7295)

Bumps [tui](https://github.com/fdehau/tui-rs) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/fdehau/tui-rs/releases)
- [Changelog](https://github.com/fdehau/tui-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/fdehau/tui-rs/compare/v0.14.0...v0.15.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump built from 0.4.4 to 0.5.0 (#7297)

Bumps [built](https://github.com/lukaslueg/built) from 0.4.4 to 0.5.0.
- [Release notes](https://github.com/lukaslueg/built/releases)
- [Commits](https://github.com/lukaslueg/built/compare/0.4.4...0.5.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* enhancement(kafka source, kafka sink): Emit internal metrics provided by rdkafka (#5874)

* Emit internal metrics for rdkafka

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Rename KafkaStatistics -> KafkaStatisticsReceived

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Emit KafkaStatisticsReceived as counter with JSON annotation

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Fix missing fields in KafkaSinkConfig

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Flatten into statistics_interval_ms

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Document statistics_interval_ms option for kafka source/sink

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Create producer in kafka sink with KafkaStatisticsContext

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Add component docs for kafka_statistics_received_total telemetry

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Emit selected metrics from kafka statistics

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Remove trivial constructor and take statistics by reference

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Remove statistics_interval_ms config options and set kafka statistics interval to 1s by default

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Change kafka metrics from gauge to counter where appropriate

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Fix unused import

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Rerun checks for benchmark results

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Fix punctuation

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Calculate counter delta locally instead of in global recorder/layer

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Emit delta before dropping mutex guard / document behavior

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Warn and emit 0 instead of panicking on non-monotonic values

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Fix non-capitalized log message

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Fix typo

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Use atomic instead of mutex in update_counter!

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Use const initializer instead of lazy_static!

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump rust_decimal from 1.12.3 to 1.12.4 (#7301)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.3 to 1.12.4.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump bstr from 0.2.15 to 0.2.16 (#7302)

Bumps [bstr](https://github.com/BurntSushi/bstr) from 0.2.15 to 0.2.16.
- [Release notes](https://github.com/BurntSushi/bstr/releases)
- [Commits](https://github.com/BurntSushi/bstr/compare/0.2.15...0.2.16)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(external docs): Typo (#7304)

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore: Make CI tests run a little more quiet (#7269)

* Separate bench and test, uh, tests

This commit re-introduces the `--quiet` flag -- this will still output for
failing tests -- and splits up the notion of the benchmark sanity tests versus
the correctness, correctness tests.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Split benches debug tests into their own workflow

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* correct 'bench unit' name

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* feedback updates

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more elaborate caching

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Revert "more elaborate caching"

This reverts commit 9ed4fdd0b4e7ae32d59531262ba63a474668efdd.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* warning fix

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove unused target

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* enhancement(remap): Wdd `whitespace` option to `parse_key_value` function (#7292)

* Empty value

Signed-off-by: ktf <krunotf@gmail.com>

* Whitespace option

Signed-off-by: ktf <krunotf@gmail.com>

* Whitespace before

Signed-off-by: ktf <krunotf@gmail.com>

* Docs

Signed-off-by: ktf <krunotf@gmail.com>

* Replace bool with Whitespace

Signed-off-by: ktf <krunotf@gmail.com>

* chore(deps): bump tracing-subscriber from 0.2.17 to 0.2.18 (#7298)

Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.17 to 0.2.18.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.17...tracing-subscriber-0.2.18)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump snap from 1.0.4 to 1.0.5 (#7300)

Bumps [snap](https://github.com/BurntSushi/rust-snappy) from 1.0.4 to 1.0.5.
- [Release notes](https://github.com/BurntSushi/rust-snappy/releases)
- [Commits](https://github.com/BurntSushi/rust-snappy/compare/1.0.4...1.0.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: Run cargo-flake on the project (#7307)

* Run cargo-flake on the project

I have struggled in #7240 to get reliable test results. This has inspired both
PR #7269 and #7306 to reduce both output and turn-around time for tests. In this
PR I introduce a tool that is meant to repeatedly run our tests and identify any
that are flakey.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* adjustments

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* `cargo flake` not `cargo-flake`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* remove temporary PR run

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore: Introduce more aggressive caching (#7306)

* Introduce more aggressive caching

This commit introduces caching of `target/debug` into our jobs as well as the
use of `restore-keys` to hit cache even when `Cargo.lock` has changed. This
commit is further work from #7269

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* `target/debug` not `~/target/debug`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Back off caching `target/debug`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* faster macOS dev builds (#7317)

Signed-off-by: Lee Benson <lee@leebenson.com>

* chore: Extract `Event` into vector-core (#7240)

* chore: Extract `Event` into vector-core

This PR extracts `Event`, `metrics` and `mapping` module into vector-core. I had
hoped to do these separately but they proved to be tied together in a way I
couldn't separate. I've intentionally not made many structural changes to this
code. As much as possible it's a simple move operation.

Closes #7151 #7148

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* address clippy warnings

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* address failing vector-core tests

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* clippy fixes

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* correct log_schema

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* correct panic! without formatting string

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* address build issue after rebase

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(deps): bump rust_decimal from 1.12.4 to 1.13.0 (#7320)

Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.4 to 1.13.0.
- [Release notes](https://github.com/paupino/rust-decimal/releases)
- [Commits](https://github.com/paupino/rust-decimal/commits/1.13.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump ordered-float from 2.1.1 to 2.2.0 (#7321)

Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 2.1.1 to 2.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v2.1.1...v2.2.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Move `benches/lookup` into vector-core (#7331)

While the fixture data for this benchmark was moved into `vector-core` as a part
of #7240 the benchmark itself was not. This broke the build.

Resolves #7329

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* fix(ci): Flag benchmark runs that fail to actually execute (#7330)

I had included `continue-on-error` originally since we didn't want to
flag regressions yet, but this also suppresses other errors (like the
benches couldn't run).

The `|| true` in `scripts/check-criterion-output.sh` should stop it from
flagging regressions as failed CI checks still.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(splunk_hec source): Default `host` to remote address (#7293)

* Use remote address

Signed-off-by: ktf <krunotf@gmail.com>

* Add test

Signed-off-by: ktf <krunotf@gmail.com>

* Add issue link

Signed-off-by: ktf <krunotf@gmail.com>

* fix(cli): Enable use of delimiters (#7316)

Signed-off-by: ktf <krunotf@gmail.com>

* enhancement(datadog sinks): Generic site support (#7281)

Add a `site` option for Datadog sinks, it aims to ultimately replace 
the `region` option.

Signed-off-by: prognant <pierre.rognant@datadoghq.com>

* chore: Reference RFC template in RFC readme (#7344)

* chore: Reference RFC template in RFC readme

Signed-off-by: Pablo Sichert <mail@pablosichert.com>

* Use relative link

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>
Signed-off-by: Pablo Sichert <mail@pablosichert.com>

Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* Tidy up iterator logic (#7338)

This commit tidies up the iterator logic suggested by clippy to be more
direct. Changed suggested by @bruceg in #7240.

Partially resolves #7327

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Make vrl optional in vector-core (#7335)

While VRL is essential to vector it is not a part of core. Allowing core to be
built without linking to VRL was originally a feature of #7240 but was removed
as it increased the testing surface overmuch.

By default, now, vector-core will not link with VRL.

Partially addresses #7327

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore: Use 'cargo hack' instead of check-component-features (#7311)

* Use 'cargo hack' instead of check-component-features

This commit replaces our check-component-features with 'cargo hack' to the same
purpose. We also remove the nightly check as it appears that it is recently
fixed.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Correct makefile typo

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Address build failure

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* kafka build issue resolved, all-metrics feature flag

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* re-introduce 'check-component-features' target

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Exclude 'utils' features from hack check

Turns out we have features that are only for utility purposes, perfectly valid
but if we check them alone we'll end up with export warnings.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(deps): bump syn from 1.0.71 to 1.0.72 (#7334)

Bumps [syn](https://github.com/dtolnay/syn) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/1.0.71...1.0.72)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump regex from 1.4.6 to 1.5.3 (#7319)

Bumps [regex](https://github.com/rust-lang/regex) from 1.4.6 to 1.5.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.6...1.5.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): bump tracing from 0.1.25 to 0.1.26 (#7296)

Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.25 to 0.1.26.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.25...tracing-0.1.26)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* fix(ci): Run tests for all targets on all platforms (#7323)

Previous benches job was actually running unit tests too. This reverts to running all targets in one job.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* Derive `Default` for `MakeLogEvent` (#7339)

As noted by @bruceg in #7240 we could avoid the explicit call to
`LogEvent::default` by deriving `Default` for `MakeLogEvent`. This is now done
and I've tidied up the relevant call-site.

Partial resolves #7327

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* chore(sinks): Drop unnecessary `Request` parameter from types (#7251)

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* chore: Drop `futures01` (#7343)

Signed-off-by: ktf <krunotf@gmail.com>

* chore(deps): bump sha-1 from 0.9.4 to 0.9.5 (#7357)

Bumps [sha-1](https://github.com/RustCrypto/hashes) from 0.9.4 to 0.9.5.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha-1-v0.9.4...sha-1-v0.9.5)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix the postinst conditionnal for adding vector to the systemd-journal group (#7359)

Signed-off-by: Johann Queuniet <sub_code.git@queuniet.fr>

* chore(deps): bump metrics from 0.14.2 to 0.15.1 (#7313)

* chore(deps): bump metrics from 0.14.2 to 0.15.1

Bumps [metrics](https://github.com/metrics-rs/metrics) from 0.14.2 to 0.15.1.
- [Release notes](https://github.com/metrics-rs/metrics/releases)
- [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml)
- [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.14.2...metrics-v0.15.1)

Signed-off-by: dependabot[bot] <support@github.com>

* Upgrade other metrics dependencies and fix conflicts

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* Update metrics versions in vector core too

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jesse Szwedko <jesse@szwedko.me>

* chore(deps): Update grok and dependencies (#7176)

* chore(deps): Update grok and dependencies

This updates grok to 1.1 and its dependencies.

I couldn't upgrade to 1.2 due to a conflict with lucet on `userfaultfd`:

```
error: failed to select a version for `userfaultfd`.
    ... required by package `lucet-runtime-internals v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
    ... which is depended on by `lucet-runtime v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
    ... which is depended on by `lucet-wasi v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)`
    ... which is depended on by `vector v0.13.0 (/Users/jesse.szwedko/workspace/vector)`
versions that meet the requirements `^0.2.0` are: 0.2.1, 0.2.0
```

This required adding additional build dependencies to the cross and Windows builds: `llvm` and `clang`.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore: Drop redundant copy of built info (#7365)

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* chore: RFC for emitting multiple log events from remap (#7038)

* chore: RFC for emitting multiple log events from remap

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore: Enable stricter clippy lints in vector-core (#7341)

* Enable stricter clippy lints in vector-core

In the early drafts of #7240 we had strict clippy lints on. While these were
desirable they ended up bloating the PR. We now re-introduce those strict
lints.

Partially resolves #7327

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* A small host of clippy::pedantic suggestions

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more clippy::pedentic suggestions

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more clippy::pedantic fixes

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* fix reference issue

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Back off mandatory `must_use`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Allow `clippy::match_wildcard_for_single_variants`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove site grant for `redundant_closure_for_method_calls`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Slightly adjust Number parse

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Site grant for `needless_pass_by_value`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove unused import

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove allow for `clippy::doc_markdown`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove global grant for allow_too_many_lines

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove global grant for enum_glob_use

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Disallow global grant for cast_possible_truncation

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove global grant for `missing_error_docs`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Remove global grant for `missing_panic_docs`

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* clippy dings in lua-town

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* add documentation

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* clippy dings in test code

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* reverse non_ascii_literal

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* documentation, 'cargo hack' detected test failures

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Document clippy lint

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more documentation

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Update lib/vector-core/src/config/log_schema.rs

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

Co-authored-by: Bruce Guenter <bruce@untroubled.org>

* Don't use approximate constants

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Value -> Self

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* address clippy ding

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

Co-authored-by: Bruce Guenter <bruce@untroubled.org>

* chore(metrics): Remove emits & alias `processed_events_total` metric (#7345)

* Add alias

Signed-off-by: ktf <krunotf@gmail.com>

* Remove emit

Signed-off-by: ktf <krunotf@gmail.com>

* Update docs

Signed-off-by: ktf <krunotf@gmail.com>

* Format cue

Signed-off-by: ktf <krunotf@gmail.com>

* chore(deps): bump regex from 1.5.3 to 1.5.4 (#7374)

Bumps [regex](https://github.com/rust-lang/regex) from 1.5.3 to 1.5.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.3...1.5.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(ci): Install llvm on Windows nightly and release bulids too (#7377)

Installed in test.yml as part of
https://github.com/timberio/vector/pull/7176

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* chore(metrics): Fix alias for processed_events_total (#7378)

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* fix(observability)!: Correct all `*_nanoseconds` metrics to `_seconds` (#7373)

* fix(observability): Correct all `*_nanoseconds` metrics to `_seconds`

All the referenced metrics labelled as "nanoseconds" are passed a
`Duration` as the value, which the `metrics` crate automatically
converts into (fractional) seconds internally. As such, all these names
are incorrect and need correction.

* Fix some more named `*_ns` but were really seconds too

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* chore(data model): Add metadata support for end-to-end acknowledgements (#7141)

* Drop unreferenced struct from RFC

* chore(data model): Add metadata for event finalization

* Add the event finalizers to the top-level metadata

* Move cloning into registration functions

This avoids extraneous use of `Arc::clone` when the item is already in
the table.

* Add a "counting" DashMap to provide serialization semantics

* Back out support for serialzing finalization status

In discussion with Brian and Luke, we decided to punt for now on
supporting end-to-end acknowledgements after an event has been buffered
to disk. This greatly simplifies the initial requirements, but we will
come back to this later.

* Drop remaining `impl From<LogEvent>` that silently drop metadata

* Quiet a clippy warning

In `enum BufferInputCloner`, the `Disk` variant is quite a bit larger
than `Memory` with the increased size of `Event` (233 vs 25 bytes).
Clippy recommends boxing this, but that is beyond the scope of this PR.

* Add `metadata_mut` accessors to event types

* Rename `EventStatus::NoOp` to `Recorded`

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* enhancement(http sink): Add support for end-to-end acknowledgements (#7265)

* Create `EventRef` wrapper for references to event inner types

* Rework MetricsBuffer to use the Metric type internally

* Merge some common bits in http sink tests

* Add tests for retries and failures to http sink

* Add Batch wrapper to store metadata

* Use metadata batch for batch sinks

* Update event metadata in `ServiceSink`

* Add metadata output to `trait HttpSink::encode_event`

* Handle metadata in `BatchSink` wrapper

* Modify http sink to pass along metadata

* Add batch status asserts to all HTTP tests

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>

* chore(ci): Fix deb verify steps (#7385)

They verify installation in a container without systemd running so the
`systemctl daemon-reload` step fails.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* feat(remap transform): Emitting multiple events (#7267)

Add support to the `remap` transform to emit multiple events by setting `.` to an array.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>

* enhancement(remap): Add parse_klog function (#7349)

Closes: #6793

This is very similar to parse_glog but I think a separate function is
still prudent in-case the continue to diverge.

The biggeste differences are just spacing and the fact that timestamps
don't have a year in the…
@spencergilbert
Copy link
Contributor

I've been looking at this today, and as far as I can tell the test doesn't wait/tail the restarted pod properly

spencergilbert
spencergilbert reviewed Jun 7, 2021
View reviewed changes
lib/k8s-e2e-tests/tests/vector-agent.rs Show resolved Hide resolved
lib/k8s-e2e-tests/tests/vector-agent.rs Outdated Show resolved Hide resolved
lib/k8s-e2e-tests/tests/vector-agent.rs Outdated Show resolved Hide resolved
@spencergilbert spencergilbert mentioned this pull request Jun 8, 2021
Merged
jszwedko
jszwedko approved these changes Jun 9, 2021
View reviewed changes
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

I think what you have here is good to merge. Ideally we might use a container where we can control its output so that we could cause it to output directly before and after the vector pod restart rather than relying on the sleep timing. Something like https://hub.docker.com/r/hashicorp/http-echo/

@spencergilbert
Copy link
Contributor

Nice work!

I think what you have here is good to merge. Ideally we might use a container where we can control its output so that we could cause it to output directly before and after the vector pod restart rather than relying on the sleep timing. Something like https://hub.docker.com/r/hashicorp/http-echo/

We'd have to add some additional bits to expose it, but that's definitely a possibility. I agree that making sure we have enough control over the tests is important

@spencergilbert
Copy link
Contributor

Waiting to merge this until I've resolved the e2e matrix issue

@jszwedko jszwedko linked an issue Jun 11, 2021 that may be closed by this pull request
Add test for checkpoints to the Kubernetes E2E test suite #4047
Closed
eeyun and others added 2 commits June 11, 2021 13:57
@eeyun @spencergilbert
Adds tests for checkpointing regressions in k8s
6963c32 
Signed-off-by: Ian Henry <ianjhenry00@gmail.com>
@spencergilbert
Identified problem with checkpoint test, unable to properly time out …
c652576 
…the test now

Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com>
@spencergilbert
Rewrote and fixed checkpoint test
13c1845 
Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com>
@spencergilbert spencergilbert merged commit aca912d into master Jun 14, 2021
@spencergilbert spencergilbert deleted the eeyun/checkpoint branch June 14, 2021 19:25
@jszwedko
Copy link
Member

🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spencergilbert spencergilbert spencergilbert left review comments

@jszwedko jszwedko jszwedko approved these changes

@StephenWakely StephenWakely Awaiting requested review from StephenWakely

Assignees

@StephenWakely StephenWakely

@spencergilbert spencergilbert

Labels
ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add test for checkpoints to the Kubernetes E2E test suite
5 participants
@eeyun @jszwedko @StephenWakely @spencergilbert @JeanMertz