Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Migrate container registry config storage from Etcd to PostgreSQL #1917

Merged
merged 103 commits into from
Sep 30, 2024
Merged

feat: Migrate container registry config storage from Etcd to PostgreSQL #1917

merged 103 commits into from
Sep 30, 2024

Conversation

jopemachine
Copy link
Member

@jopemachine jopemachine commented Feb 19, 2024 •
edited by achimnol
Loading

Resolves #1907.

Overview

  • Migrate container registry schema and data from Etcd to PostgreSQL
  • Add is_global column to the ContainerRegistry table
  • This PR does not modify the GraphQL schema, but just migrates the storage of container registry configurations.
  • /config/container-registries API is now an alias to /resource/container-registries.

Rollback if migration fails

Run backend.ai mgr etcd put-json 'config/docker/registry' backup.etcd.container-registries.{timestamp}.json to restore the original container registry configurations in etcd.
If you run the migration multiple times with failures, there will be multiple backup files. Check the file size to see if the contnet is NOT empty before rollback.

Fore more details, see #2882.

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version
  • Mention to the original issue
  • Installer updates including:
    • Fixtures for db schema changes
    • New mandatory config options
  • Test case(s) to:
    • Demonstrate the difference of before/after
    • Demonstrate the flow of abstract/conceptual models with a concrete implementation

@github-actions github-actions bot added comp:manager Related to Manager component size:XL 500~ LoC labels Feb 19, 2024
@jopemachine jopemachine force-pushed the feat/migration-container-registries branch from 51b64c8 to 497bc32 Compare February 19, 2024 02:01
@jopemachine jopemachine changed the title feat: Migrate container registry info from etcd to postgresql Migrate container registry schema from etcd to postgreSQL Feb 19, 2024
@jopemachine jopemachine linked an issue Feb 19, 2024 that may be closed by this pull request
Rebase container registry config storage to relational database #1907
Closed
@jopemachine jopemachine added this to the 24.03 milestone Feb 19, 2024
@jopemachine jopemachine changed the title Migrate container registry schema from etcd to postgreSQL feat: Migrate container registry schema from etcd to postgreSQL Feb 19, 2024
@jopemachine jopemachine force-pushed the feat/migration-container-registries branch 8 times, most recently from ecff04f to d8a29c4 Compare February 21, 2024 04:50
@jopemachine jopemachine marked this pull request as ready for review February 21, 2024 06:06
@jopemachine jopemachine added the require:db-migration Automatically set when alembic migrations are added or updated label Feb 22, 2024
@jopemachine jopemachine force-pushed the feat/migration-container-registries branch 4 times, most recently from e6d9ef0 to 127221c Compare February 29, 2024 07:10
kyujin-cho
kyujin-cho requested changes Mar 4, 2024
View reviewed changes
Copy link
Member

@kyujin-cho kyujin-cho left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current implementation does not have a flag field at container_registry table which should indicate if this registry is available for every users. Please update the branch accordingly.

src/ai/backend/manager/models/base.py Outdated Show resolved Hide resolved
@jopemachine jopemachine marked this pull request as draft March 6, 2024 04:42
@jopemachine jopemachine marked this pull request as ready for review March 7, 2024 05:33
@fregataa fregataa self-requested a review March 8, 2024 05:12
fregataa
fregataa requested changes Mar 8, 2024
View reviewed changes
Copy link
Member

@fregataa fregataa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments about typing

src/ai/backend/manager/models/image.py Outdated Show resolved Hide resolved
src/ai/backend/manager/container_registry/harbor.py Outdated Show resolved Hide resolved
src/ai/backend/manager/container_registry/harbor.py Outdated Show resolved Hide resolved
src/ai/backend/manager/container_registry/base.py Show resolved Hide resolved
@jopemachine jopemachine force-pushed the feat/migration-container-registries branch from 9891e05 to 9fd2e2a Compare March 8, 2024 06:48
@jopemachine jopemachine force-pushed the feat/migration-container-registries branch from bfd11a3 to 412b9a1 Compare September 30, 2024 12:20
@kyujin-cho kyujin-cho added this pull request to the merge queue Sep 30, 2024
Merged via the queue into main with commit f07b789 Sep 30, 2024
24 checks passed
@kyujin-cho kyujin-cho deleted the feat/migration-container-registries branch September 30, 2024 14:21
@github-actions GitHub Actions
Copy link

Backport to 24.03 is failed. Please backport manually.

@kyujin-cho kyujin-cho modified the milestones: 24.03, 24.09 Sep 30, 2024
@achimnol achimnol mentioned this pull request Sep 30, 2024
Merged
2 tasks
This was referenced Oct 1, 2024
Closed
Closed
Closed
Closed
Closed
This was referenced Oct 11, 2024
Merged
Merged
@rapsealk rapsealk mentioned this pull request Oct 17, 2024
Closed
6 tasks
lizable pushed a commit to lablup/backend.ai-webui that referenced this pull request Oct 21, 2024
@agatha197
feat: fetch container registries with new nodes query (#2746)
ccb9a9b 
- related PR: lablup/backend.ai#1917, lablup/backend.ai#2888

**TR;DR**
New GraphQL queries and mutations have been added related to the Registry since 24.09. Internally, the management from etcd has been changed to DB table. Therefore, version management is as follows:
- 24.09.0 and above: Use `ContainerRegistryList.tsx`, `ContainerRegistryEditorModal.tsx`
- 23.09.2 and above but below 24.09.0: Use `ContainerRegistryListBefore2409.tsx`, `ContainerRegistryEditorModalBefore2409.tsx`
- Below 23.09.2: Use `backend-ai-registry-list.ts`

**What's changed**
- `ContainerRegistryList.tsx` and `ContainerRegistryEditorModal.tsx` have been renamed to `ContainerRegistryListBefore2409.tsx`, `ContainerRegistryEditorModalBefore2409.tsx`.
- From 24.09.0, `ContainerRegistryList.tsx` and `ContainerRegistryEditorModal.tsx` use `container_registry_nodes` related query and mutation.
- Pagination and sorting (only for registry name) are supported.
- Updated schema

**How to test**
1. Change https://github.com/lablup/backend.ai-webui/blob/7163812c4b3a70679b0efb347e11b52c3f9948cb/react/src/pages/EnvironmentPage.tsx#L65-L73 to like this.
```ts
isSupportContainerRegistryGraphQL ? (
    <ContainerRegistryList />
) : (
````
2. Change this line https://github.com/lablup/backend.ai-webui/blob/7163812c4b3a70679b0efb347e11b52c3f9948cb/react/src/components/ContainerRegistryList.tsx#L87 to like this. (since 24.09.0 is not released yet).
```ts
          ) @SInCE(version: "24.09.0b1") {
```
3. Go to Environments - Registries page.

**Checklist for reviewers**
- [ ] CRUD registries.
- [ ] The project is set to 'library' when the type is 'docker' and the input field is disabled.
- [ ] Registry name filtering and sorting work well.
- [ ] Pagination works well.
  - For testing this, you can add '1' option to this line. https://github.com/lablup/backend.ai-webui/blob/a7ab1ed6fe540070010f87a437f52c1c42b30abc/react/src/components/ContainerRegistryList.tsx#L347

**Checklist:** (if applicable)

- [x] Mention to the original issue
- [ ] Documentation
- [x] Minium required manager version
- [x] Specific setting for review (eg., KB link, endpoint or how to setup)
- [x] Minimum requirements to check during review
- [ ] Test case(s) to demonstrate the difference of before/after
This was referenced Oct 23, 2024
Merged
Merged
Merged
Merged
Merged
@achimnol achimnol mentioned this pull request Nov 4, 2024
Open
@fregataa fregataa mentioned this pull request Nov 15, 2024
Merged
1 task
This was referenced Jan 8, 2025
Open
Open
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fregataa fregataa fregataa requested changes

@kyujin-cho kyujin-cho kyujin-cho requested changes

@achimnol achimnol Awaiting requested review from achimnol

Assignees

@jopemachine jopemachine

Labels
comp:client Related to Client component comp:installer Related to Installer comp:manager Related to Manager component require:db-migration Automatically set when alembic migrations are added or updated size:XL 500~ LoC
Projects
None yet
Milestone
24.09
Development

Successfully merging this pull request may close these issues.

Rebase container registry config storage to relational database
4 participants
@jopemachine @kyujin-cho @achimnol @fregataa