Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[system tests] Validate transforms based on the mappings #2347

Open
wants to merge 56 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
2c3f76e
Add dynamic templates as parameter
mrodm Dec 12, 2024
41d7b88
Add validation for each dynamic template depending on the parameters
mrodm Dec 16, 2024
ff5298e
Fixes for dynamic templates
mrodm Dec 16, 2024
af04d66
Compare fields with dynamic templates
mrodm Dec 17, 2024
f21c712
Remove multi_fields from flattened fields
mrodm Dec 17, 2024
aa28db6
Test without filtering dynamic templates
mrodm Dec 17, 2024
3767111
Ensure properties subfields are validated accordingly
mrodm Dec 18, 2024
9fb80b4
Restore filtering and add tests
mrodm Dec 18, 2024
2ad28ac
Disable unmatch_mapping_type and match_mapping_type and continue look…
mrodm Dec 19, 2024
8bdad65
Merge upstream/main into validate-dynamic-mappings
mrodm Dec 19, 2024
151e59e
Refactors and remove log statements
mrodm Dec 19, 2024
d0ff6b9
Fix function naming
mrodm Dec 19, 2024
b474b4b
Add test with match_pattern regex
mrodm Dec 19, 2024
066b5a0
Add comment in tests
mrodm Dec 19, 2024
a7eb191
Remove loading schema from create validator for mappings method
mrodm Jan 7, 2025
7666ac2
Separate parsing and validation of dynamic templates
mrodm Jan 7, 2025
a718796
Support match_pattern for the other settings
mrodm Jan 7, 2025
0a5e775
fix test
mrodm Jan 8, 2025
131cc9c
Update tests for multi-fields
mrodm Jan 8, 2025
38db8df
Review multi-fields logic
mrodm Jan 8, 2025
9ff3d0c
Revisit multi-fields logic in ECS
mrodm Jan 8, 2025
5165612
Report all errors related to multi-fields comparing with ECS
mrodm Jan 8, 2025
ce557c9
Ignored validation of multi-fields with ECS
mrodm Jan 8, 2025
ebda859
Fix multi-field test
mrodm Jan 8, 2025
019ffd4
Rephrase errors
mrodm Jan 9, 2025
8f36c18
Validate fully dynamic objects (preview) with dynamic templates
mrodm Jan 9, 2025
d96ffbf
Rephrase debug message
mrodm Jan 9, 2025
dbca4fe
Add logging - to be removed
mrodm Jan 9, 2025
8fce0ec
Merge remote-tracking branch 'upstream/main' into validate-dynamic-ma…
mrodm Jan 14, 2025
aa5674f
Validate transforms with mappings getting docs
mrodm Jan 15, 2025
4bf6104
Take into account deleted docs for transforms
mrodm Jan 16, 2025
9acd95c
Update logs
mrodm Jan 16, 2025
a5b635d
Force to update timestamp in ti_anomali test package
mrodm Jan 16, 2025
6fc1c2e
Add options struct for waitForDocs
mrodm Jan 16, 2025
bcca589
Update missing system test config with tags
mrodm Jan 16, 2025
0a241a9
Update ingest pipeline for ti_anomali
mrodm Jan 17, 2025
9a0b677
Sanitize values read for dynamic templates
mrodm Jan 17, 2025
ebdd98c
Move check for empty strings to elasticsearch functions
mrodm Jan 20, 2025
4d1724e
Merge upstream/main into validate-transforms-mappings
mrodm Jan 21, 2025
b9917fd
Remove unused field
mrodm Jan 21, 2025
9251224
Update stop condition in transforms
mrodm Jan 21, 2025
c50b94d
Update sync.time.field setting in transform - ti_anomali
mrodm Jan 21, 2025
ffcad91
Deleted undesired empty lines
mrodm Jan 21, 2025
460b420
Fix stop conditions to wait for docs in transforms
mrodm Jan 22, 2025
1cbe763
Add transform name to test case
mrodm Jan 23, 2025
930bbbe
Extract function to validate transforms based on mappings
mrodm Jan 23, 2025
e2eee93
Remove TODO
mrodm Jan 23, 2025
a178a68
Use processed docs as stop condition in transforms
mrodm Jan 23, 2025
aff903b
Remove unused function
mrodm Jan 23, 2025
c385cb2
Add exception for transforms in stacks < 8.14.0
mrodm Jan 24, 2025
df0732a
Merge upstream/main into validate-transforms-mappings
mrodm Jan 24, 2025
a65efc3
Check for errors in docs when validation based on mappings is used
mrodm Jan 27, 2025
1e5a7ae
Rename function
mrodm Jan 27, 2025
afac6f3
Report any mapping error or any doc with an error
mrodm Jan 27, 2025
e2fe72f
Merge upstream/main into validate-transforms-mappings
mrodm Jan 28, 2025
1d539ee
Use source data stream hits as a reference for transforms
mrodm Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions internal/elasticsearch/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,16 @@ func (c *Client) SimulateIndexTemplate(ctx context.Context, indexTemplateName st
return nil, nil, fmt.Errorf("error unmarshaling mappings: %w", err)
}

// In case there are no dynamic templates, set an empty array
if string(preview.Template.Mappings.DynamicTemplates) == "" {
preview.Template.Mappings.DynamicTemplates = []byte("[]")
}

// In case there are no mappings defined, set an empty map
if string(preview.Template.Mappings.Properties) == "" {
preview.Template.Mappings.Properties = []byte("{}")
}

return preview.Template.Mappings.DynamicTemplates, preview.Template.Mappings.Properties, nil
}

Expand Down Expand Up @@ -403,5 +413,15 @@ func (c *Client) DataStreamMappings(ctx context.Context, dataStreamName string)
mappingsDefinition = v.Mappings
}

// In case there are no dynamic templates, set an empty array
if string(mappingsDefinition.DynamicTemplates) == "" {
mappingsDefinition.DynamicTemplates = []byte("[]")
}

// In case there are no mappings defined, set an empty map
if string(mappingsDefinition.Properties) == "" {
mappingsDefinition.Properties = []byte("{}")
}

return mappingsDefinition.DynamicTemplates, mappingsDefinition.Properties, nil
}
3 changes: 3 additions & 0 deletions internal/packages/packages.go
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,9 @@ type TransformDefinition struct {
Source struct {
Index []string `config:"index" yaml:"index"`
} `config:"source" yaml:"source"`
Dest struct {
Index string `config:"index" yaml:"index"`
} `config:"dest" yaml:"dest"`
Meta struct {
FleetTransformVersion string `config:"fleet_transform_version" yaml:"fleet_transform_version"`
} `config:"_meta" yaml:"_meta"`
Expand Down
Loading