Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prod cluster noobaa-default-backing-store out of default 50Gi storage #222

Closed
computate opened this issue Sep 8, 2023 · 5 comments · Fixed by OCP-on-NERC/nerc-ocp-config#458
Closed

Prod cluster noobaa-default-backing-store out of default 50Gi storage #222

computate opened this issue Sep 8, 2023 · 5 comments · Fixed by OCP-on-NERC/nerc-ocp-config#458
Assignees
@larsks @computate
Labels
observability openshift This issue pertains to NERC OpenShift

Comments

@computate
Copy link
Member

It looks like the NooBaa default backing store is in Rejected state, probably because it NooBaa in prod has a tiny 50Gi disk that is 99% full. See these links:
@computate computate added the openshift This issue pertains to NERC OpenShift label Sep 8, 2023
@computate computate self-assigned this Sep 8, 2023
@computate
Copy link
Member Author

@Milstein mentioned for his project "It was working well until it didn't. We have ~240GB of data, and I was able to move 95.833 GiB into OpenShift. Then things stopped working. At some point, I started getting 500 errors saying InternalError: We encountered an internal error. Please try again. For more debugging info, I tried using aws s3 CLI instead of rclone and got the same result."

@computate
Copy link
Member Author

@computate mentioned "I'm also having problems writing to object storage: mc: <ERROR> Failed to copy /opt/forecast_example/template.job. We encountered an internal error. Please try again."

@computate
Copy link
Member Author

Anybody know if we should:

  • increase the noobaa default backing store storage? It's storage class is storageClass: ocs-external-storagecluster-ceph-rbd ?
  • add a new backing store to the existing noobaa bucket class?

I was trying to find the most relevant documentation, and it might be this document on tweaking object storage, and it might be this one for deploying OpenShift container storage on Google Cloud.

@larsks larsks self-assigned this Sep 27, 2023
@computate
Copy link
Member Author

Will 10Ti be enough for users in Prod for a new object storage Backing Store?

computate added a commit to computate/nerc-ocp-config that referenced this issue Jun 26, 2024
@computate
Adding 10TiB object storage to prod for user projects
ec8ccbe 
The default noobaa-default-backing-store was created with too small
(1 50Gi volume, max 20 50Gi volumes = 1Ti). We will create a larger
object-backing-store with 1 10Ti volume up to 20.

Closes nerc-project/operations#222
@computate computate mentioned this issue Jun 26, 2024
Merged
@computate
Copy link
Member Author

Now NERC OpenShift users can create Object Bucket Claims like this in their projects, pointed to the new storageClassName: object-bucket-storage.

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: my-bucket
  namespace: my-project
spec:
  generateBucketName: my-bucket
  storageClassName: object-bucket-storage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@larsks larsks

@computate computate

Labels
observability openshift This issue pertains to NERC OpenShift
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adding 10TiB object storage to prod for user projects computate/nerc-ocp-config
3 participants
@larsks @computate @joachimweyl