Skip to content

Commit

Permalink
added mechanisms for mapping secondary data folder and individual sec…
Browse files Browse the repository at this point in the history
…rets files
  • Loading branch information
mfhepp committed May 10, 2024
1 parent ecfa3a4 commit a4259f8
Show file tree
Hide file tree
Showing 7 changed files with 346 additions and 87 deletions.
3 changes: 2 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.git
.github
.secrets
docs
test
LICENSE
Expand All @@ -11,4 +12,4 @@ default-stack.md
versions.md
output/
src/output/
src/data/
src/data/
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Project
.secrets
archive
todo

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down
42 changes: 42 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,9 @@ It is **strongly recommended to use an absolute path in the alias** (otherwise,
You can build isolated containers with Juypter Notebook and JupyterLab.
**Note:** This functionality is likely to become a separate project, see [Issue 15](https://github.com/mfhepp/py4docker/issues/15)
### Building a Notebook Image
#### Using the default environment file `notebook.yaml`
Expand Down Expand Up @@ -390,6 +393,45 @@ nbh openai
nbh foo
```
#### Mapping a secondary data directory to `/mnt/data`
You can map any other directory from your system as read-only bind volume to `/mnt/data` inside the Docker container like so:
```bash
# /home/foo/bar will be accessible as /mnt/data inside the container:
./run_notebook.sh --data-dir /home/foo/bar
```
#### Mapping API tokens and other secrets from local files to `/mnt/secrets/`
You can map one or more local files containing access tokens as a read-only bind mounts to `/mnt/secrets/` inside the Docker container like so:
```bash
./run_notebook.sh --add-secret ~/Documents/.access_tokens/TESTTOKEN1 FOO \
--add-secret ~/Documents/.access_tokens/TESTTOKEN2 BAR
```
You will then be able to access them inside the notebook like so:
```bash
# Inside a notebook cell, run Bash commands with a ! directive;
!cat /mnt/secrets/FOO
!cat /mnt/secrets/BAR
# Contents of the two files TESTTOKEN1 and TESTTOKEN2
SUPERSECRET_TOKEN1
API_TOKEN_FOR_ACME
```
A Python example is in [examples/secrets_test.ipynb](examples/secrets_test.ipynb).
**Warnings:**
1. This is a ***simplistic*** substitute for Docker's mechanisms for managing secrets, but IMO more secure than using environment variables that may be leaked in logfiles etc. **Keep in mind that in the current version, ALL files inside that directory will be available from inside the container!**
2. Make sure **that you DO NOT LEAK YOUR SECRETS TO YOUR Git repository**.
3. Make sure **THAT YOUR SECRETS folder is NOT below your current working directory.** Otherwise, it will be accessible for read- and write-access from within `/usr/app/data` (e.g. as `/usr/app/data/.access_tokens/`)!!!
4. On OSX, do not use `~/.access_tokens`, but rather `~/Documents/.access_tokens`, `~/Documents/.access_tokens`, or any place in the predefined subfolders below the user directory, because
1. OSX grants ANY user on your machine read-access to any user's home folder.
2. The OSX permissions model for applications will ask you only if an application tries to access one of the specific folders ***below the user directory***. I.e., any application COULD READ from `/Users/yourusername/.access_tokens`!!!
## Advanced Topics
### Access to the Local File System
Expand Down
146 changes: 146 additions & 0 deletions examples/secrets_test.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "3c697b0f-4fa4-4dd0-a86b-90b5ed0fbaaa",
"metadata": {},
"source": [
"# Example: Accessing API tokens and credentials\n",
"\n",
"## Starting Jupyter\n",
"\n",
"Start Jupyter with\n",
"\n",
"```bash\n",
"./run_notebook.sh --add-secret ~/.my_access_tokens/TESTTOKEN1 FOO --add-secret ~/.my_access_tokens/TESTTOKEN2 BAR\n",
"```\n",
"\n",
"The two **files** \n",
"\n",
"`~/.my_access_tokens/TESTTOKEN1`\n",
"\n",
"and\n",
"\n",
"`~/.my_access_tokens/TESTTOKEN2`\n",
"\n",
"must exist on the host systems!\n"
]
},
{
"cell_type": "markdown",
"id": "91d1053e-31d8-43a0-9c53-819f9f15f7cc",
"metadata": {},
"source": [
"## Accessing the contents of the secrets from a notebook"
]
},
{
"cell_type": "markdown",
"id": "11af385e-37bf-4d72-adce-2e0e9caa8489",
"metadata": {},
"source": [
"### With shell commands"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "92b59d9d-6bc2-40ca-afb7-abb7903f7dc8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SUPERSECRET_TOKEN1\n",
"API_TOKEN_FOR_ACME\n"
]
}
],
"source": [
"!cat /mnt/secrets/FOO\n",
"!cat /mnt/secrets/BAR"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "67300fec-69df-42af-b962-5eab6799cd0b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 16\n",
"drwxr-xr-x 2 root root 4096 May 10 11:43 .\n",
"drwxr-xr-x 1 root root 4096 May 10 11:43 ..\n",
"-rw-r--r-- 1 root root 19 May 10 11:32 BAR\n",
"-rw-r--r-- 1 root root 19 May 10 11:32 FOO\n"
]
}
],
"source": [
"!ls -la /mnt/secrets/"
]
},
{
"cell_type": "markdown",
"id": "29b93823-53ea-4dc8-98bc-077440b8b54a",
"metadata": {},
"source": [
"### Python"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cde13d39-2d0a-4cba-992d-2e9e06511d4d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SUPERSECRET_TOKEN1\n",
"API_TOKEN_FOR_ACME\n"
]
}
],
"source": [
"SECRET_1_PATH = \"/mnt/secrets/FOO\"\n",
"SECRET_2_PATH = \"/mnt/secrets/BAR\"\n",
"\n",
"with open(SECRET_1_PATH, \"r\") as file:\n",
" SECRET_TOKEN = file.read().strip()\n",
"\n",
"with open(SECRET_2_PATH, \"r\") as file:\n",
" API_KEY_ACME = file.read().strip()\n",
"\n",
"print(SECRET_TOKEN)\n",
"print(API_KEY_ACME)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
29 changes: 14 additions & 15 deletions notebook.yaml.lock
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,12 @@ dependencies:
- importlib_metadata=7.1.0=hd8ed1ab_0
- importlib_resources=6.4.0=pyhd8ed1ab_0
- ipykernel=6.29.3=pyhd33586a_0
- ipython=8.22.2=pyh707e725_0
- ipywidgets=8.1.2=pyhd8ed1ab_0
- ipython=8.24.0=pyh707e725_0
- ipywidgets=8.1.2=pyhd8ed1ab_1
- isodate=0.6.1=pyhd8ed1ab_0
- isoduration=20.11.0=pyhd8ed1ab_0
- jedi=0.19.1=pyhd8ed1ab_0
- jinja2=3.1.3=pyhd8ed1ab_0
- jinja2=3.1.4=pyhd8ed1ab_0
- json5=0.9.25=pyhd8ed1ab_0
- jsonpointer=2.4=py312h996f985_3
- jsonschema=4.22.0=pyhd8ed1ab_0
Expand All @@ -68,7 +68,7 @@ dependencies:
- jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
- jupyterlab_server=2.27.1=pyhd8ed1ab_0
- jupyterlab_widgets=3.0.10=pyhd8ed1ab_0
- jupytext=1.16.1=pyhd8ed1ab_0
- jupytext=1.16.2=pyhd8ed1ab_1
- keyutils=1.6.1=h4e544f5_0
- krb5=1.21.2=hc419048_0
- ld_impl_linux-aarch64=2.40=hba4e955_0
Expand All @@ -77,17 +77,17 @@ dependencies:
- libedit=3.1.20191231=he28a2e2_2
- libexpat=2.6.2=h2f0025b_0
- libffi=3.4.2=h3557bc0_5
- libgcc-ng=13.2.0=he277a41_6
- libgfortran-ng=13.2.0=he9431aa_6
- libgfortran5=13.2.0=h87d9d71_6
- libgomp=13.2.0=he277a41_6
- libgcc-ng=13.2.0=he277a41_7
- libgfortran-ng=13.2.0=he9431aa_7
- libgfortran5=13.2.0=h87d9d71_7
- libgomp=13.2.0=he277a41_7
- libiconv=1.17=h31becfc_2
- liblapack=3.9.0=22_linuxaarch64_openblas
- libnsl=2.0.1=h31becfc_0
- libopenblas=0.3.27=pthreads_h5a5ec62_0
- libsodium=1.0.18=hb9de7d4_1
- libsqlite=3.45.3=h194ca79_0
- libstdcxx-ng=13.2.0=h3f4de04_6
- libstdcxx-ng=13.2.0=h3f4de04_7
- libuuid=2.38.1=hb4cce97_0
- libxcrypt=4.4.36=h31becfc_1
- libxml2=2.12.6=h3091e33_2
Expand All @@ -106,7 +106,7 @@ dependencies:
- nbconvert-core=7.16.4=pyhd8ed1ab_0
- nbconvert-pandoc=7.16.4=hd8ed1ab_0
- nbformat=5.10.4=pyhd8ed1ab_0
- ncurses=6.4.20240210=h0425590_0
- ncurses=6.5=h0425590_0
- nest-asyncio=1.6.0=pyhd8ed1ab_0
- notebook=7.1.3=pyhd8ed1ab_0
- notebook-shim=0.2.4=pyhd8ed1ab_0
Expand Down Expand Up @@ -134,7 +134,7 @@ dependencies:
- ptyprocess=0.7.0=pyhd3deb0d_0
- pure_eval=0.2.2=pyhd8ed1ab_0
- pycparser=2.22=pyhd8ed1ab_0
- pygments=2.17.2=pyhd8ed1ab_0
- pygments=2.18.0=pyhd8ed1ab_0
- pyparsing=3.1.2=pyhd8ed1ab_0
- pyshacl=0.25.0=pyhd8ed1ab_0
- pysocks=1.7.1=pyha2e5f31_6
Expand All @@ -146,8 +146,8 @@ dependencies:
- python_abi=3.12=4_cp312
- pytz=2024.1=pyhd8ed1ab_0
- pyyaml=6.0.1=py312hdd3e373_1
- pyzmq=26.0.2=py312h7059f03_0
- qtconsole-base=5.5.1=pyha770c72_0
- pyzmq=26.0.3=py312h7059f03_0
- qtconsole-base=5.5.2=pyha770c72_0
- qtpy=2.4.1=pyhd8ed1ab_0
- rdflib=7.0.0=pyhd8ed1ab_0
- readline=8.2=h8fc344f_1
Expand All @@ -156,7 +156,7 @@ dependencies:
- rfc3339-validator=0.1.4=pyhd8ed1ab_0
- rfc3986-validator=0.1.1=pyh9f0ad1d_0
- rich=13.7.1=pyhd8ed1ab_0
- rpds-py=0.18.0=py312h2a5c9d6_0
- rpds-py=0.18.1=py312heb99873_0
- send2trash=1.8.3=pyh0d859eb_0
- setuptools=69.5.1=pyhd8ed1ab_0
- shellingham=1.5.4=pyhd8ed1ab_0
Expand All @@ -168,7 +168,6 @@ dependencies:
- terminado=0.18.1=pyh0d859eb_0
- tinycss2=1.3.0=pyhd8ed1ab_0
- tk=8.6.13=h194ca79_0
- toml=0.10.2=pyhd8ed1ab_0
- tomli=2.0.1=pyhd8ed1ab_0
- tornado=6.4=py312h9ef2f89_0
- traitlets=5.14.3=pyhd8ed1ab_0
Expand Down
Loading

0 comments on commit a4259f8

Please sign in to comment.