Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Doc Handling] AWS S3 document store implementation #24541

Closed
14 tasks
chillleader opened this issue Nov 8, 2024 · 0 comments · Fixed by #25054
Closed
14 tasks

[Doc Handling] AWS S3 document store implementation #24541

chillleader opened this issue Nov 8, 2024 · 0 comments · Fixed by #25054
Assignees
@mo-okocha
Labels
component/document-handling Marks an issue as related to document handling kind/feature Categorizes an issue or PR as a feature, i.e. new behavior version:8.7.0-alpha3

Comments

@chillleader
Copy link
Member

chillleader commented Nov 8, 2024 •
edited by mo-okocha
Loading

Description

As part of the Document Handling capability, we need to provide the Self-Managed Camunda 8 users with a way to store their documents in a custom S3 bucket.

The store must implement all operations:

  • Create/get/delete documents
  • Generate public signed URLs
  • Verify document hash

Breakdown

  1. Implement a new DocumentStore for AWS S3
  • Test setup: Create a static test bucket for testing & development @chillleader
    • Permissions: Write objects, read objects, create pre-signed URLs, delete objects. No access to bucket parameters or lifecycle
    • Lifecycle policies need to be set
  • Align with the Controller team to ensure we create a bucket in each cluster - this happens for GCP but we will need to do the same for S3
  • Implement the document store operations
  • Cover the implementation with unit tests
  • Include the new document store into the Camunda distribution package so that it's available in the Zeebe gateway
  • Test the configuration logic and make sure it is possible to configure the S3 document store with environment variables (related: [Doc Handling] Multiple store support #24508)
  • Documentation: How can customers configure their buckets to work with Camunda? Mainly lifecycle policies & permissions
  • Align with the Distro team regarding configuration of this feature in Helm charts. How will authentication work? Default credentials chain must be supported @chillleader

See GCP DocumentStore for reference
2. Align with QA to test the new store implementation
3. Document the new feature in https://github.com/camunda/camunda-docs
@chillleader chillleader added kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. component/zeebe Related to the Zeebe component/team component/document-handling Marks an issue as related to document handling kind/feature Categorizes an issue or PR as a feature, i.e. new behavior and removed component/zeebe Related to the Zeebe component/team kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. labels Nov 8, 2024
@mo-okocha mo-okocha mentioned this issue Nov 22, 2024
Merged
3 tasks
github-merge-queue bot pushed a commit that referenced this issue Dec 16, 2024
@mo-okocha
feat: implement aws document store (#25054)
eb9ccdb 
## Description

<!-- Describe the goal and purpose of this PR. -->
This update introduces the ability to use AWS S3 as the storage platform
for documents managed through the Documents API.

#### Environment variables
`DOCUMENT_STORE_AWS_BUCKET_NAME` => The name of the AWS S3 bucket used
for document storage.

`DOCUMENT_STORE_AWS_BUCKET_PATH` => The path within the S3 bucket for
storing documents (e.g., `test/bucket/path/`).

`DOCUMENT_STORE_AWS_BUCKET_TTL` => The time-to-live (TTL) for document
objects in the AWS S3 bucket. This value is specified in days and
defines when the objects will be automatically deleted. Do not set to
disable TTL.

#### Features covered
- Uploading documents to S3
- Fetching documents from S3
- Deleting documents from S3
- Generating pre-signed URL for documents
- Expired documents return document not found

#### Features not covered
- Hash Verification

## Checklist

<!--- Please delete options that are not relevant. Boxes should be
checked by reviewer. -->
- [ ] for CI changes:
- [ ] structural/foundational changes signed off by [CI
DRI](https://github.com/cmur2)
- [ ]
[ci.yml](https://github.com/camunda/camunda/blob/main/.github/workflows/ci.yml)
modifications comply with ["Unified CI"
requirements](https://github.com/camunda/camunda/wiki/CI-&-Automation#workflow-inclusion-criteria)

## Related issues

closes #24541
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mo-okocha mo-okocha

Labels
component/document-handling Marks an issue as related to document handling kind/feature Categorizes an issue or PR as a feature, i.e. new behavior version:8.7.0-alpha3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: implement aws document store camunda/camunda
3 participants
@camundait @chillleader @mo-okocha