chore(config): initial integration of configuration schema for sources #13005
Conversation
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
… bits Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
… polluting cargo output Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
tobz
added
the
ci-condition: integration tests enable
✅ Deploy Preview for vector-project canceled.
|
github-actions
bot
added
domain: core
| vector_config = { path = "lib/vector-config" } | ||
| vector_config_macros = { path = "lib/vector-config-macros" } |
There was a problem hiding this comment.
As a note: for some reason, the compiler is not happy with me re-exporting the macros from vector_config_macros in vector_config as a way to only need to add vector_config to a given crate.
While doing re-exports in this way seems to work fine for the other dependencies in the public API of vector_config::Configurable (schemars, indexmap, etc), it gets angry about the procedural macros and complains that the crate is missing, etc, in an error at the actual callsite i.e. wherever #[configurable_component] is being used.
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Soak Test ResultsBaseline: d417613 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Soak Test ResultsBaseline: 8bccadb ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
There was a problem hiding this comment.
This is coming along nicely!
For other reviewers, you can run cargo test vector_config -- --nocapture to see the generated schema.
I skimmed over most of the doc comments and focused more on the generated schema which looks good to me. I expect it'll drift if we modify the cue docs, but is a big step in the right direction. Hopefully we can drop the CUE definitions sooner than later (maybe we could generate the CUE definition from the JSON schema until the docs switch over?).
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Soak Test ResultsBaseline: dfc2e29 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
Soak Test ResultsBaseline: 116d0b3 ExplanationA soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core. The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed. No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%: Fine details of change detection per experiment.
|
(part of the work required for #12223)
This PR is the initial implementation of the configuration schema framework, specifically for sources.
It does not change how we load configurations, or anything about our use of
inventory/typetag/SourceOuter/etc. That work will be the final step after all components have been minimally instrumented/converted to support having their configuration schema generated.Reviewer Notes
In general, the a small portion of this PR is adding the
#[configurable_component]attribute macro to the configuration types for sources, including any types that they transitively depend on. Additionally, due to a requirement by theConfigurablederive, all derived types must have public documentation (sometimes specified on fields directly, sometimes derived from the field type itself) which represents the vast majority of actual changed lines in this PR.I've strived to port over the existing Cue documentation in a faithful manner, but there are a number of small divergences:
host_keyortls.*, etc) consistency across sources