Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a parameter to the CFN stack for S3 changelog to disable accumulating the events #86

Merged
merged 2 commits into from
May 16, 2024
Merged

Add a parameter to the CFN stack for S3 changelog to disable accumulating the events #86

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e86d988
Add parameter to disable event rule and refresh
volatilecat May 16, 2024
16f0933
Add a warning
volatilecat May 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions elastio-s3-changelog/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Then, the Elastio `iscan` job reads those events to perform the scanning of new

1. First, you need to enable Amazon EventBridge for your S3 buckets by following these instructions:
[Enabling Amazon EventBridge](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications-eventbridge.html).

2. Use one of the following quick-create links. Choose the region where your Elastio Cloud Connector is deployed.

**Important!** You can change the stack name, but it **MUST** start with `elastio-`. Otherwise, Elastio won't be able to access the created resources.
Expand All @@ -32,13 +33,16 @@ Then, the Elastio `iscan` job reads those events to perform the scanning of new

* *ScanExistingObjects* - set to `true` if you want to perform the initial scan of all objects in the bucket(s);

* *KeyPrefixes* - comma-separated list of prefixes of objects to scan. This will be applied to all buckets.
* *KeyPrefixes* - (optional) comma-separated list of prefixes of objects to scan. This will be applied to all buckets.
If you want to use different prefixes for different buckets, you need to deploy multiple S3 Changelog stacks.
Also, note that the paths selector in the Protection Policy will also be used to filter objects before scanning.
This means that the *KeyPrefixes* parameter must be in sync with the paths selector in the Protection Policy,
or not specified at all.
or not specified at all;

* *DisableEventCollection* - if you decide to disable the protection policy for your S3 bucket, you can set this
to `true` to stop accumulating S3 update events in the SQS queue.
Comment on lines +42 to +43
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* *DisableEventCollection* - if you decide to disable the protection policy for your S3 bucket, you can set this
to `true` to stop accumulating S3 update events in the SQS queue.
* *DisableEventCollection* - if you decide to disable the protection policy for your S3 bucket, you can set this
to `true` to stop accumulating S3 update events in the SQS queue.
**IMPORTANT**: If you set this parameter to `true`, then no more objects will be scanned on this bucket. If you set it back to `false`, then any new objects after the parameter is set to `false` will be available to scan, but all changes made to the bucket during the time when this parameter was set to `true` will not be scanned, unless you manually initiate a full scan of the entire bucket.


> There are also some advanced and experimental parameters in the template, you can ignore them.
> There are other advanced and experimental parameters in the template, you can ignore them.

4. Check the box in front of `I acknowledge that AWS CloudFormation might create IAM resources with custom names`
and `I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND`
Expand Down
10 changes: 7 additions & 3 deletions elastio-s3-changelog/cloudformation-multiple-buckets-with-macros.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ Parameters:
ScanExistingObjects:
Type: String
Default: 'false'
KeyPrefixes:
Type: String
DisableEventCollection:
Type: String
Default: 'false'
EnableDlq:
Type: String
Default: 'true'
Expand All @@ -25,8 +30,6 @@ Parameters:
QrtsMaxDelay:
Type: Number
MinValue: 60
KeyPrefixes:
Type: String
Resources:
'Fn::ForEach::Stacks':
- BucketName
Expand All @@ -37,11 +40,12 @@ Resources:
Parameters:
BucketName: !Ref BucketName
ScanExistingObjects: !Ref ScanExistingObjects
KeyPrefixes: !Ref KeyPrefixes
DisableEventCollection: !Ref DisableEventCollection
EnableDlq: !Ref EnableDlq
EnableQrts: !Ref EnableQrts
QrtsBatchSize: !Ref QrtsBatchSize
QrtsMaxDelay: !Ref QrtsMaxDelay
KeyPrefixes: !Ref KeyPrefixes
TemplateURL: https://{{S3_BUCKET}}.s3.{{AWS_REGION}}.amazonaws.com/{{S3_PREFIX}}/{{VERSION}}/cloudformation-single-bucket.yaml
Outputs:
templateVersion:
Expand Down
10 changes: 9 additions & 1 deletion elastio-s3-changelog/cloudformation-multiple-buckets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ Parameters:
is enabled, paths selector in the Protection Policy will also be used to filter objects before scanning.
Example: xyz/, foo/bar/

DisableEventCollection:
Type: String
AllowedValues: ['true', 'false']
Default: 'false'
Description: If set to true, disables the changelog events collection.

EnableDlq:
Type: String
AllowedValues: ['true', 'false']
Expand Down Expand Up @@ -74,6 +80,7 @@ Metadata:

- Label: { default: Advanced Parameters }
Parameters:
- DisableEventCollection
- EnableDlq

- Label: { default: Experimental Parameters }
Expand All @@ -89,11 +96,12 @@ Resources:
Parameters:
BucketNames: !Join [',', !Ref BucketNames]
ScanExistingObjects: !Ref ScanExistingObjects
KeyPrefixes: !Join [',', !Ref KeyPrefixes]
DisableEventCollection: !Ref DisableEventCollection
EnableDlq: !Ref EnableDlq
EnableQrts: !Ref EnableQrts
QrtsBatchSize: !Ref QrtsBatchSize
QrtsMaxDelay: !Ref QrtsMaxDelay
KeyPrefixes: !Join [',', !Ref KeyPrefixes]
TemplateURL: https://{{S3_BUCKET}}.s3.{{AWS_REGION}}.amazonaws.com/{{S3_PREFIX}}/{{VERSION}}/cloudformation-multiple-buckets-with-macros.yaml

Outputs:
Expand Down
17 changes: 16 additions & 1 deletion elastio-s3-changelog/cloudformation-single-bucket.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ Parameters:
is enabled, paths selector in the Protection Policy will also be used to filter objects before scanning.
Example: xyz/,foo/bar/

DisableEventCollection:
Type: String
AllowedValues: ['true', 'false']
Default: 'false'
Description: If set to true, disables the changelog events collection.

EnableDlq:
Type: String
AllowedValues: ['true', 'false']
Expand Down Expand Up @@ -73,6 +79,7 @@ Metadata:

- Label: { default: Advanced Parameters }
Parameters:
- DisableEventCollection
- EnableDlq

- Label: { default: Experimental Parameters }
Expand All @@ -83,6 +90,7 @@ Metadata:

Conditions:
ScanExistingObjects: !Equals [!Ref ScanExistingObjects, 'true']
DisableEventCollection: !Equals [!Ref DisableEventCollection, 'true']
EnableDlq: !Equals [!Ref EnableDlq, 'true']
EnableQrts: !Equals [!Ref EnableQrts, 'true']
NoKeyPrefixes: !Equals [!Ref KeyPrefixes, '']
Expand Down Expand Up @@ -125,6 +133,10 @@ Resources:
Type: AWS::Events::Rule
Properties:
Description: !Sub Track S3 object change events in ${BucketName}
State: !If
- DisableEventCollection
- DISABLED
- ENABLED
EventPattern: !If
- NoKeyPrefixes
- !Sub >
Expand Down Expand Up @@ -209,7 +221,10 @@ Resources:
ScheduleExpression: cron(0/15 * * * ? *)
FlexibleTimeWindow:
Mode: 'OFF'
State: ENABLED
State: !If
- DisableEventCollection
- DISABLED
- ENABLED
Target:
Arn: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:elastio-s3-changelog-handler-refresh'
RoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/ElastioS3ChangelogSchedulerExecution'
Expand Down