-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release 4.9.0 - RC 2 - Vulnerability Detection performance test #25507
Comments
Moved to
Relaunch of medium case due to failure in the graphs and very high activity case due it has failed: |
Medium ActivityArtifacts: artifacts.zip Comparision: Compared with #24504 Build: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/666/ Logs 🟡SummaryMaster 🟡
Worker 1 🟡
Worker 2 🟡
Indexer 1 🟡
Indexer 2 🟡
Dashboard 🟢
Metrics and Statistics 🟡Master 🟡
|
High ActivityBuild: CLUSTER-Workload_benchmarks_metrics/667/ Logs 🟡Master 🟡2024/09/02 15:40:54 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2024/09/02 15:40:54 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-10-253.ec2.internal', retrying until the connection is successful. 2024/09/02 15:54:50 wazuh-remoted: ERROR: (1320): Agent '000' not found. Worker 1 🟡
2024/09/02 15:55:41 wazuh-analysisd: WARNING: Syscollector decoder queue is full.
2024/09/02 16:26:24 wazuh-remoted: WARNING: Message queue is full (131072). Events may be lost. 2024/09/02 15:40:53 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2024/09/02 15:40:53 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-1-233.ec2.internal', retrying until the connection is successful. Worker 2 🟡
2024/09/02 15:55:44 wazuh-analysisd: WARNING: Syscollector decoder queue is full.
2024/09/02 15:56:00 wazuh-remoted: WARNING: Message queue is full (131072). Events may be lost. 2024/09/02 15:40:53 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2024/09/02 15:40:53 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-2-238.ec2.internal', retrying until the connection is successful. 2024/09/02 15:55:40 wazuh-remoted: WARNING: Package dropped. Could not append data into buffer.
2024/09/02 15:55:40 wazuh-remoted: WARNING: (1246): Unable to send file 'merged.mg' to agent ID '092'. Indexer 0 🟡
[2024-09-02T15:36:57,477][WARN ][o.o.s.c.Salt ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-09-02T15:36:57,516][WARN ][o.o.s.a.r.AuditMessageRouter] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] No default storage available, audit log may not work properly. Please check configuration
[2024-09-02T15:36:58,351][WARN ][o.o.s.p.SQLPlugin ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information.
[2024-09-02T15:36:59,426][WARN ][o.o.g.DanglingIndicesState] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-09-02T15:37:01,488][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Config override setting update called with empty string. Ignoring.
[2024-09-02T15:36:59,474][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Fail to read queue capacity via reflection [2024-09-02T15:37:04,494][WARN ][o.o.p.c.u.JsonConverter ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"]) [2024-09-02T15:36:57,515][ERROR][o.o.s.a.s.SinkProvider ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Default endpoint could not be created, auditlog will not work properly. [2024-09-02T15:36:59,471][ERROR][o.o.p.c.j.GCMetrics ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] MX bean missing: G1 Concurrent GC [2024-09-02T15:37:02,420][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security) Indexer 1 🟡
[2024-09-02T15:36:57,647][WARN ][o.o.s.c.Salt ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-09-02T15:36:57,681][WARN ][o.o.s.a.r.AuditMessageRouter] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] No default storage available, audit log may not work properly. Please check configuration.
[2024-09-02T15:36:58,547][WARN ][o.o.s.p.SQLPlugin ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-09-02T15:37:01,475][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Config override setting update called with empty string. Ignoring.
[2024-09-02T15:37:02,084][WARN ][o.o.o.i.ObservabilityIndex] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] message: index [.opensearch-observability/kes8L5raQf623sRwI7YZNg] already exists
[2024-09-02T15:39:59,964][WARN ][o.o.s.c.ConfigurationRepository] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Unable to reload configuration, initalization thread has not yet completed.
[2024-09-02T15:36:59,280][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Fail to read queue capacity via reflection [2024-09-02T15:37:04,291][WARN ][o.o.p.c.u.JsonConverter ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"]) [2024-09-02T15:36:57,680][ERROR][o.o.s.a.s.SinkProvider ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Default endpoint could not be created, auditlog will not work properly. [2024-09-02T15:36:59,293][ERROR][o.o.p.c.j.GCMetrics ] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] MX bean missing: G1 Concurrent GC [2024-09-02T15:37:02,513][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [CLUSTER-Workload_benchmarks_metrics_B667_indexer_1] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security) Dashboard 🟢No logs Metrics and Statistics 🟡Master 🟡
Worker 1 🟢
Continued at #25507 (comment) |
Medium activityMetrics and statistics 🟡Worker 1 🟡
Worker 2 🟡
Vulnerability State 🟢Alerts 🟢 |
High ActivityMetrics and Statistics 🟡Worker 2 🟡
Vulnerabilities State 🟢Alerts 🟢
Start at #25507 (comment) |
ETA increase due to relaunch of very high and medium activity cases. |
Update
|
LGTM |
Tests information
Description
The objective is to conduct performance tests to analyze the vulnerability detection module across varying environment loads: high, medium, and low. This comparative analysis will provide a comprehensive understanding of the current status of the vulnerability detection module and will help identify any unexpected behavior.
Methodology
Utilizing the CLUSTER-Workload_benchmarks_metrics pipeline to execute specified test cases automatically. Results will be manually analyzed and shared with the development team for validation adjustments.
Test Cases
Considerations
KIBANA_API_REQUESTS
should be enabledEXTRA_LOAD_API_REQUESTS
should be enabledBuilds
Conclusion 🔴
Known Issues 🟡
Fail to read queue capacity via reflection
warning in Indexer log wazuh-indexer#71MX bean missing: G1 Concurrent GC
error message wazuh-indexer#328New issue 🔴
The text was updated successfully, but these errors were encountered: