adding ignore errors and retries for continue on error/fail on error

[zackbradys]

adding strict mode for continue on error/fail on error

Please check below, if the PR fulfills these requirements:

• Commit(s) and code follow the repositories guidelines.
• Test(s) have been added or updated to support these change(s).
• Doc(s) have been added or updated to support these change(s).

Associated Links:

• resolves... [feature] Possible Way To Ignore Hauler Store Sync Errors? #349
• resolves... [feature] Return a non 0 exit code on error #323
• partially resolves... [feature] sync or push should continue after error #296
• partially resolves... [feature] retries for copy operations #284

Types of Changes:

• Feature

Proposed Changes:

• Added a store flag of --retries/-r that allows users to set the number of retries when fetching artifacts
• Added a root flag of --ignore-errors and environment variable of HAULER_IGNORE_ERRORS that currently enables/allows users to have hauler warn upon encountering an error

Verification/Testing of Changes:

• N/A

Additional Context:

• N/A

[zackbradys]

without --ignore-errors

zackbradys@Zacks-MacBook-Pro hauler % ./dist/hauler_darwin_arm64_v8.0/hauler store sync --files testdata/hauler-manifest.yaml           
2024-11-30 09:45:20 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images] to store
2024-11-30 09:45:20 INF adding 'image' [busybox] to the store
2024-11-30 09:45:29 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2024-11-30 09:45:29 INF adding 'image' [busybox:stables] to the store
2024-11-30 09:45:30 ERR error (attempt 1/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:45:35 ERR error (attempt 2/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:45:41 ERR error (attempt 3/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:45:41 ERR unable to add 'image' [index.docker.io/library/busybox:stables] to store: operation unsuccessful after 3 attempts
Error: operation unsuccessful after 3 attempts
Usage:
  hauler store sync [flags]

Flags:
  -f, --files strings             Location of content manifests (files)... i.e. --files ./rke2-files.yaml
  -h, --help                      help for sync
  -k, --key string                (Optional) Location of public key to use for signature verification
  -p, --platform string           (Optional) Specify the platform of the image... i.e linux/amd64 (defaults to all)
  -c, --product-registry string   (Optional) Specify the product registry. Defaults to RGS Carbide Registry (rgcrprod.azurecr.us)
      --products strings          (Optional) Specify the product name to fetch collections from the product registry i.e. rancher=v2.8.5,rke2=v1.28.11+rke2r1
  -g, --registry string           (Optional) Specify the registry of the image for images that do not alredy define one

Global Flags:
  -d, --haulerdir string   Set the location of the hauler directory (default $HOME/.hauler)
      --ignore-errors      Ignore/Bypass errors (i.e. warn on error) (defaults false)
  -l, --log-level string   Set the logging level (i.e. info, debug, warn) (default "info")
  -r, --retries int        Set the number of retries for operations (default 3)
  -s, --store string       Set the directory to use for the content store (default "store")

2024-11-30 09:45:41 ERR operation unsuccessful after 3 attempts

with ignore-errors

zackbradys@Zacks-MacBook-Pro hauler % ./dist/hauler_darwin_arm64_v8.0/hauler store sync --ignore-errors  --files testdata/hauler-manifest.yaml
2024-11-30 09:45:59 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images] to store
2024-11-30 09:45:59 INF adding 'image' [busybox] to the store
2024-11-30 09:46:09 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2024-11-30 09:46:09 INF adding 'image' [busybox:stables] to the store
2024-11-30 09:46:09 WRN warning (attempt 1/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:46:15 WRN warning (attempt 2/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:46:20 WRN warning (attempt 3/3)... getting image "index.docker.io/library/busybox:stables": GET https://index.docker.io/v2/library/busybox/manifests/stables: MANIFEST_UNKNOWN: manifest unknown; unknown tag=stables
2024-11-30 09:46:20 WRN unable to add 'image' [index.docker.io/library/busybox:stables] to store: operation unsuccessful after 3 attempts... skipping...
2024-11-30 09:46:20 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2024-11-30 09:46:23 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2024-11-30 09:46:23 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts] to store
2024-11-30 09:46:23 INF adding 'chart' [rancher] to the store
2024-11-30 09:46:24 INF successfully added 'chart' [hauler/rancher:2.8.5]
2024-11-30 09:46:24 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.0
Digest: sha256:7b3524029fee18c886b56c174d5a558e9b7ecdf464372a2aee48fe1da9128cee
2024-11-30 09:46:25 INF successfully added 'chart' [hauler/hauler-helm:1.1.0]
2024-11-30 09:46:25 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files] to store
2024-11-30 09:46:25 INF adding 'file' [https://get.rke2.io] to the store as [hauler/install.sh:latest]
2024-11-30 09:46:25 INF successfully added 'file' [hauler/install.sh:latest]
2024-11-30 09:46:25 INF adding 'file' [testdata/hauler-manifest.yaml] to the store as [hauler/hauler-manifest.yaml:latest]
2024-11-30 09:46:25 INF successfully added 'file' [hauler/hauler-manifest.yaml:latest]