Change docker compose workflow to not require tar.gz download #1505

cgrinds · 2022-11-17T14:54:25Z

No description provided.

cgrinds · 2023-02-08T12:59:35Z

We investigated this and nothing to pursue right now. Closing

cgrinds · 2023-02-09T13:29:37Z

Investigate if this can be improved in 23.05

rahulguptajss · 2023-08-03T07:56:32Z

Moving to next release. For this release we made improvements via #2046

rahulguptajss · 2023-09-07T06:37:36Z

Assigning to myself for relevant changes.

rahulguptajss · 2023-09-07T11:48:37Z

Code Changes
Documentation
Updated CI/CD to use the documented commands

Hardikl · 2023-11-09T09:01:47Z

Tested with nightly image. Case 2 is failing so re-opening the issue.

case1 : with harvest.yml in current folder

root@localhost:~/dockertest# pwd
/root/dockertest
root@localhost:~/dockertest# ls -ltr
total 4
-rw-r--r-- 1 root root 1562 Nov  9 03:11 harvest.yml
root@localhost:~/dockertest# docker run --rm \
>   --entrypoint "bin/harvest" \
>   --volume "$(pwd):/opt/temp" \
>   --volume "$(pwd)/harvest.yml:/opt/harvest/harvest.yml" \
>   ghcr.io/netapp/harvest:nightly \
>   generate docker full \
>   --output harvest-compose.yml
Unable to find image 'ghcr.io/netapp/harvest:nightly' locally
nightly: Pulling from netapp/harvest
63b450eae87c: Pull complete 
960043b8858c: Pull complete 
b4ca4c215f48: Pull complete 
eebb06941f3e: Pull complete 
02cd68c0cbf6: Pull complete 
d3c894b5b2b0: Pull complete 
b40161cd83fc: Pull complete 
46ba3f23f1d3: Pull complete 
4fa131a1b726: Pull complete 
4a2d7bc46f58: Pull complete 
f826710fbc51: Pull complete 
Digest: sha256:c05ca3d24b9afb7efbf3eeb2ff32ef3813df98524a5fc9b4494491d4085783f9
Status: Downloaded newer image for ghcr.io/netapp/harvest:nightly
Wrote file_sd targets to container/prometheus/harvest_targets.yml
Start containers with:
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
root@localhost:~/dockertest# ls -ltr
total 28
-rw-r--r--  1 root root 1562 Nov  9 03:11 harvest.yml
drwxr-x---  2 root root 4096 Nov  9 03:12 cert
drwxr-x--- 10 root root 4096 Nov  9 03:12 conf
drwxr-xr-x  3 root root 4096 Nov  9 03:12 container
-rw-------  1 root root 1529 Nov  9 03:12 prom-stack.yml
-rw-------  1 root root 1881 Nov  9 03:12 harvest-compose.yml
drwxr-x---  4 root root 4096 Nov  9 03:12 grafana
root@localhost:~/dockertest# docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
Creating network "harvest_backend" with the default driver
Creating network "harvest_frontend" with the default driver
Creating volume "harvest_prometheus_data" with default driver
Creating volume "harvest_grafana_data" with default driver
Pulling umeng-aff300 (ghcr.io/netapp/harvest:latest)...
latest: Pulling from netapp/harvest
dd5ad9c9c29f: Pull complete
960043b8858c: Pull complete
b4ca4c215f48: Pull complete
eebb06941f3e: Pull complete
02cd68c0cbf6: Pull complete
d3c894b5b2b0: Pull complete
b40161cd83fc: Pull complete
46ba3f23f1d3: Pull complete
4fa131a1b726: Pull complete
4a2d7bc46f58: Pull complete
905eaa3efc35: Pull complete
Digest: sha256:f21e787bd33e885bd1fb3180496e1755271601df43f47da19b69ed9e3cc6d06d
Status: Downloaded newer image for ghcr.io/netapp/harvest:latest
Pulling prometheus (prom/prometheus:v2.33.1)...
v2.33.1: Pulling from prom/prometheus
3cb635b06aa2: Pull complete
34f699df6fe0: Pull complete
55d40b50ffc7: Pull complete
73fc1407aa37: Pull complete
28be42bd9809: Pull complete
16a1311431d0: Pull complete
6e6dfbb21ff4: Pull complete
cec290d8c550: Pull complete
3504f0a4fc48: Pull complete
459aec2a59be: Pull complete
838b4d86e3f3: Pull complete
6bb61ef5ec02: Pull complete
Digest: sha256:3a75763d209af6ef82aca8fb202a76092a690d25f24d2def7f4eff64e79b7ed9
Status: Downloaded newer image for prom/prometheus:v2.33.1
Pulling grafana (grafana/grafana:8.3.4)...
8.3.4: Pulling from grafana/grafana
59bf1c3509f3: Pull complete
dc6fd4c8b7cc: Pull complete
921cda16a84c: Pull complete
b2e847bc52f5: Pull complete
00a566fb83a4: Pull complete
a40995d122ce: Pull complete
dd54763fab15: Pull complete
e147c181ae77: Pull complete
cc6525936d2d: Pull complete
3cdad610e821: Pull complete
Digest: sha256:cf81d2c753c8d14cf0324d881e49f42d400347d256e473856e1bda02eb520a99
Status: Downloaded newer image for grafana/grafana:8.3.4
Creating poller-dc1          ... done
Creating poller-umeng-aff300 ... done
Creating poller-aff250       ... done
Creating poller-dc1-zapi     ... done
Creating poller-unix         ... done
Creating prometheus          ... done
Creating grafana             ... done
root@localhost:~/dockertest# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                                           NAMES
888f1b72c675   grafana/grafana:8.3.4           "/run.sh"                3 seconds ago   Up 2 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp       grafana
174e31c6347a   prom/prometheus:v2.33.1         "/bin/prometheus --c…"   8 seconds ago   Up 3 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp       prometheus
31d0415007ef   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 3 seconds   0.0.0.0:12992->12992/tcp, :::12992->12992/tcp   poller-unix
982cea9b2eaa   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12995->12995/tcp, :::12995->12995/tcp   poller-dc1-zapi
543823c3012e   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12994->12994/tcp, :::12994->12994/tcp   poller-dc1
150c39060971   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12990->12990/tcp, :::12990->12990/tcp   poller-umeng-aff300
a8a540e42e96   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12993->12993/tcp, :::12993->12993/tcp   poller-aff250

case2 : with harvest.yml in different folder

root@localhost:~/dockertest# pwd
/root/dockertest
root@localhost:~/dockertest# ls -ltr
total 0
root@localhost:~/dockertest# ls -ltr /root/harvestconfig/
total 4
-rw-r--r-- 1 root root 1562 Nov  9 03:00 harvest.yml
root@localhost:~/dockertest# HYML="/root/harvestconfig/harvest.yml" \
> docker run --rm \
> --entrypoint "bin/harvest" \
> --volume "$(pwd):/opt/temp" \
> --volume "${HYML}:${HYML}" \
> ghcr.io/netapp/harvest:nightly \
> generate docker full \
> --output harvest-compose.yml \
> --config "${HYML}"
Unable to find image 'ghcr.io/netapp/harvest:nightly' locally
nightly: Pulling from netapp/harvest
63b450eae87c: Pull complete 
960043b8858c: Pull complete 
b4ca4c215f48: Pull complete 
eebb06941f3e: Pull complete 
02cd68c0cbf6: Pull complete 
d3c894b5b2b0: Pull complete 
b40161cd83fc: Pull complete 
46ba3f23f1d3: Pull complete 
4fa131a1b726: Pull complete 
4a2d7bc46f58: Pull complete 
f826710fbc51: Pull complete 
Digest: sha256:c05ca3d24b9afb7efbf3eeb2ff32ef3813df98524a5fc9b4494491d4085783f9
Status: Downloaded newer image for ghcr.io/netapp/harvest:nightly
error reading  err=open : no such file or directory
root@localhost:~/dockertest#

rahulguptajss · 2023-11-09T15:07:23Z

@Hardikl Could you pls verify the failed step as mentioned in #2473

Hardikl · 2023-11-09T15:23:34Z

Tested with the steps mentioned in the PR #2473 , working as expected.

case2 : with harvest.yml in different folder

root@scspa2802353001:~/dockertest# pwd
/root/dockertest
root@scspa2802353001:~/dockertest# ls -ltr
total 0
root@scspa2802353001:~/dockertest# ls -ltr /root/harvestconfig/
total 4
-rw-r--r-- 1 root root 1562 Nov  9 03:00 harvest.yml
root@scspa2802353001:~/dockertest# HYML="/root/harvestconfig/harvest.yml"; \
> docker run --rm \
> --entrypoint "bin/harvest" \
> --volume "$(pwd):/opt/temp" \
> --volume "${HYML}:${HYML}" \
> ghcr.io/netapp/harvest:nightly \
> generate docker full \
> --output harvest-compose.yml \
> --config "${HYML}"
Unable to find image 'ghcr.io/netapp/harvest:nightly' locally
nightly: Pulling from netapp/harvest
63b450eae87c: Pull complete 
960043b8858c: Pull complete 
b4ca4c215f48: Pull complete 
eebb06941f3e: Pull complete 
02cd68c0cbf6: Pull complete 
d3c894b5b2b0: Pull complete 
b40161cd83fc: Pull complete 
46ba3f23f1d3: Pull complete 
4fa131a1b726: Pull complete 
4a2d7bc46f58: Pull complete 
f826710fbc51: Pull complete 
Digest: sha256:c05ca3d24b9afb7efbf3eeb2ff32ef3813df98524a5fc9b4494491d4085783f9
Status: Downloaded newer image for ghcr.io/netapp/harvest:nightly
Wrote file_sd targets to container/prometheus/harvest_targets.yml
Start containers with:
docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
root@scspa2802353001:~/dockertest# docker-compose -f prom-stack.yml -f harvest-compose.yml up -d --remove-orphans
Creating network "harvest_backend" with the default driver
Creating network "harvest_frontend" with the default driver
Creating volume "harvest_prometheus_data" with default driver
Creating volume "harvest_grafana_data" with default driver
Pulling umeng-aff300 (ghcr.io/netapp/harvest:latest)...
latest: Pulling from netapp/harvest
dd5ad9c9c29f: Pull complete
960043b8858c: Pull complete
b4ca4c215f48: Pull complete
eebb06941f3e: Pull complete
02cd68c0cbf6: Pull complete
d3c894b5b2b0: Pull complete
b40161cd83fc: Pull complete
46ba3f23f1d3: Pull complete
4fa131a1b726: Pull complete
4a2d7bc46f58: Pull complete
905eaa3efc35: Pull complete
Digest: sha256:f21e787bd33e885bd1fb3180496e1755271601df43f47da19b69ed9e3cc6d06d
Status: Downloaded newer image for ghcr.io/netapp/harvest:latest
Pulling prometheus (prom/prometheus:v2.33.1)...
v2.33.1: Pulling from prom/prometheus
3cb635b06aa2: Pull complete
34f699df6fe0: Pull complete
55d40b50ffc7: Pull complete
73fc1407aa37: Pull complete
28be42bd9809: Pull complete
16a1311431d0: Pull complete
6e6dfbb21ff4: Pull complete
cec290d8c550: Pull complete
3504f0a4fc48: Pull complete
459aec2a59be: Pull complete
838b4d86e3f3: Pull complete
6bb61ef5ec02: Pull complete
Digest: sha256:3a75763d209af6ef82aca8fb202a76092a690d25f24d2def7f4eff64e79b7ed9
Status: Downloaded newer image for prom/prometheus:v2.33.1
Pulling grafana (grafana/grafana:8.3.4)...
8.3.4: Pulling from grafana/grafana
59bf1c3509f3: Pull complete
dc6fd4c8b7cc: Pull complete
921cda16a84c: Pull complete
b2e847bc52f5: Pull complete
00a566fb83a4: Pull complete
a40995d122ce: Pull complete
dd54763fab15: Pull complete
e147c181ae77: Pull complete
cc6525936d2d: Pull complete
3cdad610e821: Pull complete
Digest: sha256:cf81d2c753c8d14cf0324d881e49f42d400347d256e473856e1bda02eb520a99
Status: Downloaded newer image for grafana/grafana:8.3.4
Creating poller-dc1-zapi     ... done
Creating poller-aff250       ... done
Creating poller-dc1          ... done
Creating prometheus          ... done
Creating poller-unix         ... done
Creating poller-umeng-aff300 ... done
Creating grafana             ... done
root@scspa2802353001:~/dockertest# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                                           NAMES
3bec6bedc83d   grafana/grafana:8.3.4           "/run.sh"                4 seconds ago   Up 2 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp       grafana
8a1428de0455   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 3 seconds   0.0.0.0:12990->12990/tcp, :::12990->12990/tcp   poller-umeng-aff300
f686bf49c0d9   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 3 seconds   0.0.0.0:12992->12992/tcp, :::12992->12992/tcp   poller-unix
abaebfc3f379   prom/prometheus:v2.33.1         "/bin/prometheus --c…"   8 seconds ago   Up 3 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp       prometheus
7872c319fff4   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12994->12994/tcp, :::12994->12994/tcp   poller-dc1
ac482332e62c   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12993->12993/tcp, :::12993->12993/tcp   poller-aff250
70be68060d74   ghcr.io/netapp/harvest:latest   "bin/poller --poller…"   8 seconds ago   Up 4 seconds   0.0.0.0:12995->12995/tcp, :::12995->12995/tcp   poller-dc1-zapi
root@scspa2802353001:~/dockertest#

cgrinds added feature New feature or request 23.02 labels Nov 17, 2022

rahulguptajss self-assigned this Nov 24, 2022

rahulguptajss added the status/open label Nov 24, 2022

cgrinds mentioned this issue Dec 12, 2022

Harvest should support docker compose workflow without requiring local tar.gz #1581

Closed

rahulguptajss removed their assignment Dec 22, 2022

cgrinds closed this as completed Feb 8, 2023

cgrinds added 23.05 and removed status/open 23.02 labels Feb 8, 2023

cgrinds reopened this Feb 9, 2023

rahulguptajss self-assigned this Apr 24, 2023

rahulguptajss added 23.08 and removed 23.05 labels Apr 26, 2023

rahulguptajss removed their assignment Aug 3, 2023

rahulguptajss added 23.11 and removed 23.08 labels Aug 3, 2023

rahulguptajss linked a pull request Aug 3, 2023 that will close this issue

doc: update Docker Compose generation process to remove binary dependencies #2046

Merged

cgrinds assigned Hardikl Aug 22, 2023

rahulguptajss assigned rahulguptajss and unassigned Hardikl Sep 7, 2023

rahulguptajss added the status/open label Sep 7, 2023

rahulguptajss linked a pull request Sep 7, 2023 that will close this issue

feat: docker workflow doesn't need tar download #2354

Merged

cgrinds closed this as completed in #2354 Sep 12, 2023

rahulguptajss removed the status/open label Sep 12, 2023

rahulguptajss added the status/testme label Sep 12, 2023

rahulguptajss linked a pull request Oct 23, 2023 that will close this issue

feat: update CI to use docker run and update permissions #2436

Merged

rahulguptajss removed their assignment Nov 6, 2023

Hardikl self-assigned this Nov 9, 2023

Hardikl reopened this Nov 9, 2023

Hardikl removed their assignment Nov 9, 2023

Hardikl added status/open and removed status/testme labels Nov 9, 2023

rahulguptajss added status/testme and removed status/open labels Nov 9, 2023

rahulguptajss closed this as completed Nov 9, 2023

rahulguptajss assigned Hardikl Nov 9, 2023

rahulguptajss added status/done and removed status/testme labels Nov 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change docker compose workflow to not require tar.gz download #1505

Change docker compose workflow to not require tar.gz download #1505

cgrinds commented Nov 17, 2022

cgrinds commented Feb 8, 2023

cgrinds commented Feb 9, 2023

rahulguptajss commented Aug 3, 2023

rahulguptajss commented Sep 7, 2023

rahulguptajss commented Sep 7, 2023

Hardikl commented Nov 9, 2023

rahulguptajss commented Nov 9, 2023

Hardikl commented Nov 9, 2023

Change docker compose workflow to not require tar.gz download #1505

Change docker compose workflow to not require tar.gz download #1505

Comments

cgrinds commented Nov 17, 2022

cgrinds commented Feb 8, 2023

cgrinds commented Feb 9, 2023

rahulguptajss commented Aug 3, 2023

rahulguptajss commented Sep 7, 2023

rahulguptajss commented Sep 7, 2023

Hardikl commented Nov 9, 2023

rahulguptajss commented Nov 9, 2023

Hardikl commented Nov 9, 2023