Deploy the same repo via kustomize to different namespaces on the same cluster

[stbraley]

My repository is structured as follows. However, when I provide a path to the dev folder. Fleet does not seem to find the base folder? I'd like to use this repo to deploy multiple instances of my app to different namespaces on the same cluster. Currently we are using ArgoCD to do this by pointing it to the overlay folder we want to deploy. For example, the dev folder. When we use Argo, it finds the base folder. In Fleet, however, I can't seem to find a way to do the same thing. I do see a way to use the same repo to deploy to multiple clusters using different overlay folders by using a fleet.yaml file. However, it does not look like the fleet.yaml file will support the same cluster with different namespaces. Am I missing something in the documentation?

Error Produced

error while running post render on files: accumulating resources: accumulateFile "accumulating resources from '../../base': '../../base' doesn't exist", loader.New "error loading ../../base with git: url lacks host: ../../base, dir: '../../base' doesn't exist, get: invalid source string: ../../base"

Repo Structure:

• base
• overlays
  • dev
    • kustomization.yaml
  • stage
  • prod
  • temp

gz#14201

[ron1]

You might try creating an individual fleet.yaml file for each overlay.

[stbraley]

Yes, I've tried. It doesn't work. It looks like Fleet requires the base directory to be a child of the path. If this is true. Then kustomize in fleet won't work for multiple instances in the same cluster. Very disappointing as this is basic 101 type stuff. If I can apply my kustomization.yaml file using the following. Fleet should be able to do the same.

kubectl apply -k overlays\dev -n dev

kustomization.yaml located in overlays\dev

bases:
- ../../base
resources:
  - ingress.yaml
patches:
  - image-tag.yaml
  - replica-count.yaml
configMapGenerator:
- name: myapp
  behavior: replace
  envs:
    - configs/settings.env

[ron1]

It seems like the default Fleet repo scanning algorithm described here: https://github.com/rancher/fleet/blob/master/docs/gitrepo-structure.md#how-repos-are-scanned should be more configurable via the GitRepo resource to allow explicit specification of one or more fleet.yaml file paths. This would allow the Fleet GitRepo resource to behave more like the ArgoCD Application resource to support this type of scenario. This would allow you to create one Fleet GitRepo resource per overlay that explicitly references a single fleet.yaml.

[ron1]

@stbraley Please provide a copy of your GitRepo manifest.

[stbraley]

I just used the UI to build the manifest. But here is what it generated.

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  creationTimestamp: "2021-01-05T20:47:40Z"
  generation: 3
  name: caas-demo
  namespace: apps7
  resourceVersion: "301964103"
  selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/apps7/gitrepos/caas-demo
  uid: 2b23e586-992a-40a8-ac01-5711c60d15d7
spec:
  branch: fleet
  clientSecretName: my-secret
  forceSyncGeneration: 2
  paths:
  - /caas/overlays/demo
  repo: repourl
  targetNamespace: demo
  targets:
  - clusterSelector:
      matchLabels:
        management.cattle.io/cluster-name: c-vzcs4
status:
  commit: 6b44fec07e1e40dfd53947bda53a48b5e88f7613
  conditions:
  - lastUpdateTime: "2021-01-05T20:50:18Z"
    message: 'ErrApplied(1) [Bundle caas-demo-caas-overlays-demo: error while running
      post render on files: accumulating resources: accumulateFile "accumulating resources
      from ''../../base'': ''../../base'' doesn''t exist", loader.New "error loading
      ../../base with git: url lacks host: ../../base, dir: ''../../base'' doesn''t
      exist, get: invalid source string: ../../base"]'
    status: "False"
    type: Ready
  - lastUpdateTime: "2021-01-05T20:50:18Z"
    status: "True"
    type: Accepted
  - lastUpdateTime: "2021-01-05T20:49:51Z"
    status: "False"
    type: Reconciling
  - lastUpdateTime: "2021-01-05T20:47:40Z"
    status: "False"
    type: Stalled
  - lastUpdateTime: "2021-01-05T20:49:51Z"
    status: "True"
    type: Synced
  desiredReadyClusters: 1
  display:
    error: true
    message: 'error while running post render on files: accumulating resources: accumulateFile
      "accumulating resources from ''../../base'': ''../../base'' doesn''t exist",
      loader.New "error loading ../../base with git: url lacks host: ../../base, dir:
      ''../../base'' doesn''t exist, get: invalid source string: ../../base"'
    readyBundleDeployments: 0/1
    state: ErrApplied
  gitJobStatus: Current
  observedGeneration: 3
  readyClusters: 0
  resourceCounts:
    desiredReady: 0
    missing: 0
    modified: 0
    notReady: 0
    orphaned: 0
    ready: 0
    unknown: 0
    waitApplied: 0
  resourceErrors:
  - 'error while running post render on files: accumulating resources: accumulateFile
    "accumulating resources from ''../../base'': ''../../base'' doesn''t exist", loader.New
    "error loading ../../base with git: url lacks host: ../../base, dir: ''../../base''
    doesn''t exist, get: invalid source string: ../../base"'
  summary:
    desiredReady: 1
    errApplied: 1
    nonReadyResources:
    - bundleState: ErrApplied
      message: 'error while running post render on files: accumulating resources:
        accumulateFile "accumulating resources from ''../../base'': ''../../base''
        doesn''t exist", loader.New "error loading ../../base with git: url lacks
        host: ../../base, dir: ''../../base'' doesn''t exist, get: invalid source
        string: ../../base"'
      name: caas-demo-caas-overlays-demo
    ready: 0
type: fleet.cattle.io.gitrepo

[ron1]

@stbraley Does the fleet example in the following git repo adequately capture your use case: https://github.com/ron1/fleet-examples/tree/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace?

@StrongMonkey Is the above example a development-centric use case Fleet would aspire to support? Today, we must use ArgoCD deployed via Fleet to support these development-centric, multi-namespace workloads.

[stbraley]

@ron1 I think so with one caveat. It needs to ignore folders in the overlay directories that don't have a path specified in the GitRepo resource. We have some deployments in the git repository that are not intended to be deployed to clusters.

[ron1]

@stbraley @StrongMonkey I made a minor change to the GitRepo in file: https://github.com/ron1/fleet-examples/blob/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace/README.md to allow multiple fleetFiles per path. A GitRepo path with one or more explicit fleetFiles that include helm and/or kustomize directives should not search for unspecified raw yaml manifests. So, I would expect Fleet to ignore "...deployments in the git repository that are not intended to be deployed to clusters".

[sridhav]

Just found a solution to this problem. may be try doing the following in your fleet.yaml to get same deployment working in multiple nameaspaces or same namespaces. this worked for me on fleet 0..6. hope this can be helpful for you guys as well

namespace: lab
targetCustomizations:
- name: dev
  helm:
    values:
      replicas: 1
  clusterSelector:
    matchLabels:
      env: dev

namespace: lab2
targetCustomizations:
- name: dev
  helm:
    values:
      replicas: 3
  clusterSelector:
    matchLabels:
      env: dev```

[e100]

We have multiple kustomize overlays in a single git repository that have many shared bases. We want to deploy each overlay from a unique GitRepo, into a specific namespace, where each GitRepo would point to a specific tag. This way we can deploy specific overlays/namespaces, one at a time, by changing the revision that items's tag points to.

The fleetFiles that @ron1 suggested would be a solution.

Using multiple paths does not work because we cannot share any common bases between the different paths.

Without some solution we cannot use fleet.

[kalihman]

Is there any progress on this issue? @ron1 implementation looks good to me. Is there anything I can contribute to make this move forward?

[manno]

For @ron1's example we would need a PR that changes the bundlereader, which is part of the CLI.
The current code takes the GitRepo.spec.paths and passes them to the fleet CLI to create bundles, like this: fleet apply --targets-file=/run/config/targets.yaml -- name path1 path2. That can be run separately.

Then CreateBundles scans each path recursively, detecting and creating "bundles" as documented in How Repos are Scanned.

I would prefer an addition to the current implementation that does not scan recursively. The detection makes Fleet easy to use, but sometimes hard to debug. Ideally I could define the bundles in the GitRepo explicitly:

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: example
spec:
  repo: ...
  bundles:
  - path1
  - path2

However, to solve this issue, we also need to be able to include another directory at the same level or below the bundle directory.

The mentioned approach with fleetFiles to specify the fleet.yaml would work. We could even put the fleet.yamls in subfolders?

.
├── base
├── overlays
│   ├── dev
│   └── prod
├── fleet1.yaml    # points to overlays/dev
├── fleet2.yaml   # points to overlays/prod

I wonder about the semantics, this would translate to:

• take path: as a base dir for the bundle's resources
• generate a bundle for every fleetFiles entry

spec:
  repo: https://github.com/rancher/fleet-examples
  paths:
  - path: multi-cluster/kustomize-multi-namespace
    fleetFiles:
    - fleet-dev-dev1.yaml
    - fleet-prod.yaml

I would prefer, to not collide with the scanner for paths: and flat over nested:

spec:
  repo: ...
  bundles:
  - path: multi-cluster/kustomize-multi-namespace
    options: fleet-dev-dev1.yaml
  - path: multi-cluster/kustomize-multi-namespace
    options: fleet-prod.yaml

fleet.yaml is eventually turned into BundleSpec and ImageScans.

[e100]

@manno For us, we would prefer to store the fleet file in a sub-folder but use the root of the repo as the base for path so things can be more organized. I'm OK with flat over nested.

.
├── base
├── overlays
    ├── dev
    |    ├── fleet.yaml # points to this overlay
    └── prod
         ├── fleet.yaml # points to this overlay

Or

├── fleetFiles
|   ├── fleet-dev-dev1.yaml # points to overlays/dev
|   ├── fleet-prod.yaml # points to overlays/prod
├── base
├── overlays
    ├── dev
    └── prod

Then deploy a specific fleetfile like:

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: dev1
  namespace: fleet-default
spec:
  repo: https://github.com/rancher/fleet-examples
  paths:
  - path: multi-cluster/kustomize-multi-namespace
    fleetFile: fleetFiles/fleet-dev-dev1.yaml

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: prod
  namespace: fleet-default
spec:
  repo: https://github.com/rancher/fleet-examples
  paths:
  - path: multi-cluster/kustomize-multi-namespace
    fleetFile: fleetFiles/fleet-prod.yaml
    ```