[Stack 8.16.0-SNAPSHOT] [crowdstrike] Failing test daily: system test: default in crowdstrike.fdr

[elastic-vault-github-plugin-prod]

• Stack version: 8.16.0-SNAPSHOT
• Package: crowdstrike
• Failing test: system test: default
• DataStream: fdr
• Owners:
  • @elastic/security-service-integrations

Failure:

test case failed: could not find hits in logs-crowdstrike.fdr-49032 data stream

First build failed: https://buildkite.com/elastic/integrations/builds/14131

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/14934
• https://buildkite.com/elastic/integrations/builds/14990
• https://buildkite.com/elastic/integrations/builds/14993
• https://buildkite.com/elastic/integrations/builds/15011
• https://buildkite.com/elastic/integrations/builds/15051

[andrewkroh]

I think this may be a result of elastic/beats#40309 which was first introduced in Elastic Agent v8.15.0.

The cause is that the SQS notification does not contain a region so the new cross-region compatibility code fails when the region is empty string. I think we need to make a check for empty region here and reuse the pre-made AWS client that derives the region from the SQS URL.

Workaround

As a temporary workaround, I suggest modifying the "FDR Notification Parsing Script" to statically set the appropriate s3 region name. I verified this in our aws integration test.

             });
             files.forEach(function(f){
               var evt = new S3EventV2();
+              evt.SetAWSRegion("us-east-1");
               evt.SetS3BucketName(bucket);
               evt.SetS3ObjectKey(f.path);
               evts.push(evt);

[andrewkroh]

From running the system test locally on v8.15.0, the error logged by the input is

{
  "log.level": "warn",
  "@timestamp": "2024-08-23T16:50:05.406Z",
  "message": "Failed processing SQS message.",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "aws-s3-default",
    "type": "aws-s3"
  },
  "log": {
    "source": "aws-s3-default"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "log.logger": "input.aws-s3",
  "log.origin": {
    "file.line": 182,
    "file.name": "awss3/sqs_input.go",
    "function": "github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*sqsReaderInput).workerLoop"
  },
  "message_id": "07c6cc02-fa16-4860-a401-367a5d47adb9",
  "elapsed_time_ns": 9348291,
  "id": "aws-s3-crowdstrike.fdr-208cffc8-3fec-4889-a98d-9c825a4e746a",
  "queue_url": "https://sqs.us-east-1.amazonaws.com/144492464627/elastic-package-crowdstrike-queue-93717",
  "error": {
    "message": "failed processing SQS message (it will return to queue after visibility timeout): failed processing S3 event for object key \"fdrv2/aidmaster\" in bucket \"elastic-package-crowdstrike-fdr-93717\" (object record 1 of 3 in SQS notification): S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.; failed processing S3 event for object key \"fdrv2/userinfo\" in bucket \"elastic-package-crowdstrike-fdr-93717\" (object record 2 of 3 in SQS notification): S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.; failed processing S3 event for object key \"data\" in bucket \"elastic-package-crowdstrike-fdr-93717\" (object record 3 of 3 in SQS notification): S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name."
  }
}

[andrewkroh]

PR for 8.16: elastic/beats#40628
PR for 8.15: elastic/beats#40632