Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bucket: Add prefix filtering capability #1228

Merged
merged 3 commits into from
Oct 17, 2023
Merged

bucket: Add prefix filtering capability #1228

merged 3 commits into from
Oct 17, 2023

Conversation

stefanprodan
Copy link
Member

@stefanprodan stefanprodan commented Sep 8, 2023

This PR adds a new optional field .spec.prefix to the Bucket API. When specified the prefix is used to filter the objects server-side when .spec.provider is set to generic, aws and gcp.

Fix: #693
Fix: #248

Testing

if you want to try out this feature on your own test cluster:

  1. Install the latest Flux controllers
flux install
  1. Apply the Bucket CRD from this branch
kubectl apply --server-side --field-manager=flux -f https://raw.githubusercontent.com/fluxcd/source-controller/bucket-prefix/config/crd/bases/source.toolkit.fluxcd.io_buckets.yaml
  1. Deploy the source-controller build from this branch
kubectl -n flux-system set image --field-manager=flux deployment/source-controller manager=ghcr.io/fluxcd/source-controller:bucket-prefix-8e215264

@stefanprodan stefanprodan added enhancement New feature or request area/bucket Bucket related issues and pull requests labels Sep 8, 2023
@tiffanywang3
Copy link

Thank you @stefanprodan; using the updated source controller image + Bucket CRD, we were able to leverage the prefix filtering which reduced Bucket reconciliation time from ~2.5 min to ~6 seconds (generic provider).

Looking forward to this update getting included in next Flux release!

Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
@stefanprodan stefanprodan marked this pull request as ready for review October 17, 2023 11:36
Signed-off-by: Stefan Prodan <stefan.prodan@gmail.com>
@stefanprodan stefanprodan added the area/api API related issues and pull requests label Oct 17, 2023
@stefanprodan stefanprodan merged commit f2a1814 into main Oct 17, 2023
11 checks passed
@stefanprodan stefanprodan deleted the bucket-prefix branch October 17, 2023 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api API related issues and pull requests area/bucket Bucket related issues and pull requests enhancement New feature or request
Projects
None yet
3 participants