service-jobs is sending auth headers to a ES AWS instance without auth in place

[xuoja-acc]

Hi,

Starting with the version 5.7.4 with the double-logging enable we have this error our logs :

2023-04-03 14:13:08.202 ERROR 1 --- [pool-6-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task
org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: "{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic Og=="}"
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:168) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:819) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:777) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) ~[spring-web-5.3.20.jar!/:5.3.20]
        at org.springframework.web.client.RestTemplate.put(RestTemplate.java:497) ~[spring-web-5.3.20.jar!/:5.3.20]
        at com.epam.reportportal.elastic.SimpleElasticSearchClient.lambda$save$1(SimpleElasticSearchClient.java:64) ~[classes!/:na]
        at java.base/java.util.HashMap.forEach(HashMap.java:1337) ~[na:na]
        at com.epam.reportportal.elastic.SimpleElasticSearchClient.save(SimpleElasticSearchClient.java:63) ~[classes!/:na]
        at com.epam.reportportal.log.LogProcessing.process(LogProcessing.java:33) ~[classes!/:na]
        at com.epam.reportportal.calculation.BatchProcessing.processAndSchedule(BatchProcessing.java:54) ~[classes!/:na]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.20.jar!/:5.3.20]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Same as : reportportal/service-auto-analyzer#121

[xuoja-acc]

Hi @IvanKustau & @APiankouski ,
Can you pay attention to this issue please?

[xuoja-acc]

Hi @rkukharenka ,

I'm sorry but it is a blocker for the usage of managed service like AWS Opensearch, is there a other topic/project to post this issue for triage / prioritization?

[DzmitryHumianiuk]

@xuoja-acc you can keep a double entry as OFF for now.

[xuoja-acc]

hi @DzmitryHumianiuk , thank's for your answer !

We didn't disable this because this documentation says double entry is a pre-requisite for a smooth upgrade to RP 5.8 :

https://reportportal.io/blog/double-entry-in-5.7.2

We recommend updating to version 5.7.2 for a smooth transition of full logging to ElasticSearch, especially if you have many logs. If you update to version 5.7.2, use it for 3-4 months before version 5.8. This period will be enough for the vast majority of projects to generate enough logs history inside ElasticSearch. And then update to version 5.8 once it is available. Since all logs will already be stored in ElasticSearch, no efforts will be required to do the migration. Along with version 5.8 we will distribute a migration script and instructions for data migration so that you can easily migrate from the early 5.x version.

[DzmitryHumianiuk]

@xuoja-acc Absolutely, you've got it right.

We implemented dual logging to eliminate the need for data migration.
Essentially, we're running two data sources in parallel—one for the old format and one for the new.
We're gradually transitioning to the new format through incremental updates while phasing out the old one.

However, we're currently reevaluating our full-scale switch to ElasticSearch, primarily due changes to their licensing model. This could potentially create issues for us down the line, something we hadn't anticipated when we initially chose this approach.

So, right now, we're exploring various alternatives to maintain our machine learning indexes without getting caught in a bind with ElasticSearch.

[xuoja-acc]

We finally managed this issue by enabling the fine-grained access-control and creating user/password.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html