Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support scanning the local Docker daemon's image list #1097

Merged
merged 20 commits into from
Feb 27, 2023

Conversation

achimnol
Copy link
Member

@achimnol achimnol commented Feb 23, 2023

resolves #1089

Note that I don't add --local option to the client CLI's admin image rescan command because this is intended for local all-in-one development setups where you can directly access the manager CLI command mgr image rescan.

Known restriction

When starting a new session via CLI, you must prepend "index.docker.io/library/" to your locally built image names if they don't have hostname/path prefixes.

Examples:

  • bai-python:3.11 -> index.docker.io/library/bai-python:3.11
  • localdev/bai-python:3.11 -> index.docker.io/localdev/bai-python:3.11

You also need to add index.docker.io to the allowed registry list of the default domain.

@achimnol achimnol added this to the 23.03 milestone Feb 23, 2023
@achimnol achimnol added type:feature Add new features comp:manager Related to Manager component comp:cli Related to CLI component labels Feb 23, 2023
@achimnol achimnol self-assigned this Feb 23, 2023
@github-actions github-actions bot added the effort:easy Need to understand only a specific region of codes (good first issue, easy). label Feb 23, 2023
- fix: Ensure the structural conformance of the ImageRow.resources column
- fix: Workaround missing manifest of locally built images by using the image ID
- Added `images.is_local` column to skip registry-related logic properly.
@achimnol achimnol marked this pull request as ready for review February 24, 2023 06:55
@achimnol achimnol added the require:db-migration Automatically set when alembic migrations are added or updated label Feb 24, 2023
@achimnol
Copy link
Member Author

achimnol commented Feb 24, 2023

Now you can build a simple Dockerfile like this:

FROM python:3.11

# Install ipython kernelspec
RUN python3.11 -m pip install ipython ipykernel
RUN python3.11 -m ipykernel install --display-name "Python 3.11 on Backend.AI" && \
    cat /usr/local/share/jupyter/kernels/python3/kernel.json

LABEL ai.backend.kernelspec=1 \
      ai.backend.features="uid-match batch query" \
      ai.backend.base-distro="ubuntu18.04" \
      ai.backend.runtime-type="python" \
      ai.backend.runtime-path="/usr/local/bin/python3"

by:

docker build -t bai-python:3.11 -f python.Dockerfile .

and run it locally on a development setup as follows:

./backend.ai mgr image rescan --local
./backend.ai mgr start-server
./backend.ai ag start-server
source env-local-admin-api.sh
./backend.ai admin domain update \  # required for the first time only
  --allowed-docker-registries cr.backend.ai \  # depends on your setup
  --allowed-docker-registries index.docker.io \
  default
./backend.ai run --rm -c 'import sys;print("hello", sys.version_info)' index.docker.io/library/bai-python:3.11

Remaining TODO: Update the existing test suite

@achimnol achimnol requested a review from kyujin-cho February 26, 2023 07:32
@achimnol
Copy link
Member Author

achimnol commented Feb 26, 2023

TODO: prevent duplication of pulled images? (maybe checking the digest in the database)

- We have been using config-digest (image ID) for other registry
  backends.  Let's just use it in the local registry impl as well.
@achimnol achimnol added this pull request to the merge queue Feb 27, 2023
Merged via the queue into main with commit 63c4469 Feb 27, 2023
@achimnol achimnol deleted the feature/local-docker-image-scan branch February 27, 2023 12:05
@Yaminyam Yaminyam added the size:L 100~500 LoC label Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:cli Related to CLI component comp:manager Related to Manager component effort:easy Need to understand only a specific region of codes (good first issue, easy). require:db-migration Automatically set when alembic migrations are added or updated size:L 100~500 LoC type:feature Add new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add backend.ai mgr image rescan --local option
2 participants