Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] needs dependency graph is broken #1881

Open
dudicoco opened this issue Jun 14, 2021 · 26 comments
Open

[BUG] needs dependency graph is broken #1881

dudicoco opened this issue Jun 14, 2021 · 26 comments

Comments

@dudicoco
Copy link
Contributor

With helmfile v0.139.9 it seems that the needs dependencies behaviour is broken.

With the following example:

- name: amazing-app-test-1
  chart: my-repo/amazing-app
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - devops/amazing-app-test-3
  values: []
- name: amazing-app-test-2
  chart: my-repo/amazing-app
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - devops/amazing-app-test-1
  values: []
- name: amazing-app-test-3
  chart: my-repo/amazing-app
  version: 2.0.14
  installed: true
  timeout: 300
  needs: []
  values: []

Previous behaviour - v0.138.4 (correct):

Upgrading release=amazing-app-test-3, chart=my-repo/amazing-app
Release "amazing-app-test-3" does not exist. Installing it now.
NAME: amazing-app-test-3
LAST DEPLOYED: Mon Jun 14 10:55:29 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-3$
amazing-app-test-3	devops   	1       	2021-06-14 10:55:29.884771 +0300 IDT	deployed	amazing-app-2.0.14	1.0

Upgrading release=amazing-app-test-1, chart=my-repo/amazing-app
Release "amazing-app-test-1" does not exist. Installing it now.
NAME: amazing-app-test-1
LAST DEPLOYED: Mon Jun 14 10:56:28 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-1$
amazing-app-test-1	devops   	1       	2021-06-14 10:56:28.275669 +0300 IDT	deployed	amazing-app-2.0.14	1.0

Upgrading release=amazing-app-test-2, chart=my-repo/amazing-app
Release "amazing-app-test-2" does not exist. Installing it now.
NAME: amazing-app-test-2
LAST DEPLOYED: Mon Jun 14 10:57:09 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-2$
amazing-app-test-2	devops   	1       	2021-06-14 10:57:09.207185 +0300 IDT	deployed	amazing-app-2.0.14	1.0

New behaviour - v0.139.9 (incorrect - releases 1 & 2 are installed in parallel):

Upgrading release=amazing-app-test-3, chart=my-repo/amazing-app
Release "amazing-app-test-3" does not exist. Installing it now.
NAME: amazing-app-test-3
LAST DEPLOYED: Mon Jun 14 11:01:38 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-3$
amazing-app-test-3	devops   	1       	2021-06-14 11:01:38.096974 +0300 IDT	deployed	amazing-app-2.0.14	1.0

Upgrading release=amazing-app-test-2, chart=my-repo/amazing-app
Upgrading release=amazing-app-test-1, chart=my-repo/amazing-app
Release "amazing-app-test-1" does not exist. Installing it now.
NAME: amazing-app-test-1
LAST DEPLOYED: Mon Jun 14 11:02:16 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-1$
Release "amazing-app-test-2" does not exist. Installing it now.
NAME: amazing-app-test-2
LAST DEPLOYED: Mon Jun 14 11:02:16 2021
NAMESPACE: devops
STATUS: deployed
REVISION: 1
TEST SUITE: None

Listing releases matching ^amazing-app-test-2$
amazing-app-test-1	devops   	1       	2021-06-14 11:02:16.553073 +0300 IDT	deployed	amazing-app-2.0.14	1.0

amazing-app-test-2	devops   	1       	2021-06-14 11:02:16.419325 +0300 IDT	deployed	amazing-app-2.0.14	1.0
@paologallinaharbur
Copy link

I noticed the very same behaviour, it seemed that the need option was completely ignored 😕

@kbbqiu
Copy link

kbbqiu commented Jul 7, 2021

Running into this as well.

@kevin-lindsay-1
Copy link

yep, I've got a pretty simple test set up on my end, and needs does nothing right now.

@dudicoco
Copy link
Contributor Author

dudicoco commented Jul 8, 2021

@mumoshu can you please address this?

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 9, 2021

Sorry everyone I've been occupied by other matters these days so I won't be able to work on this shortly.

It would be great if anyone could stand up and submit a PR for this. I'll gladly review any PR related to this ASAP.

@dm3ch
Copy link

dm3ch commented Jul 14, 2021

Maybe it's caused by this https://github.com/roboll/helmfile/releases/tag/v0.139.2? it makes default behaviour to ignore needs.

If it's so it's possible to enable needs back with --include-needs or --skip-needs=false option. But I haven't yet time to test that

@rafilkmp3
Copy link

can be enabled as a default with

helmDefaults:
  skipDeps: false

@dudicoco
Copy link
Contributor Author

dudicoco commented Jul 25, 2021

--include-needs or --skip-needs=false will not solve this issue.

@mumoshu can the whole include-needs feature get reverted in order to fix this bug?
We are currently blocked from upgrading our helmfile version.

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

@dm3ch Ah that's a good point! Maybe we should enable --include-needs by default, rather than enabling --skip-needs?

If anyone could submit a feature to change the default, like I've done for --skip-needs in #1835, I would gladly review it.

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

@dudicoco Does it have something to do with you're missing namespace: devops in your releases, even though you refer to dependencies with namespaces like devops/amazing-app-test-3? My understanding was that you can use namespacced dependency only when you have namespace defined in the depended release.

@dudicoco
Copy link
Contributor Author

@mumoshu we have one helmfile.yaml file which loads all of the releases and templates them, during the templating the namespace is added, so this is not the issue here.
This has worked for us in v0.138.4.

--include-needs will not solve the issue, as we do not want to force add releases defined in the needs block unless they were explicitly included via --label.

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

@dm3ch Well would you mind clarifying what are you expecting from --include-needs vs --skip-needs, exactly? They controls how Helmfile handles indirectly selected releases, to be installed along with directly selected releases, or to be skipped. But rereading your comment I'm not entirely sure if you understand that

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

@dudicoco Gotcha! Thanks. Do you have any blocker that prevents you from fixing Helmfile in the main branch(as I currently lack spare time to work on Helmfile) OR sticking with v0.138.4?

@dudicoco
Copy link
Contributor Author

@mumoshu can you please elaborate on what you mean by fixing Helmfile in the main branch?

On a side note, perhaps additional maintainers should be added to the repository? There is a high traffic on this repository and I understand the difficulty in maintaining it all by yourself :)

Thanks

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

can you please elaborate on what you mean by fixing Helmfile in the main branch?

@dudicoco I mean "what is preventing you from submitting a pull request yourself" 😃

On a side note, perhaps additional maintainers should be added to the repository? There is a high traffic on this repository and I understand the difficulty in maintaining it all by yourself :)

Yes, I and @renehernandez have already done our best at #1824 and a few Slack channels but had no luck so far. I don't have admin privilege on this repo so I can't add additional maintainers as of today.

@mumoshu
Copy link
Collaborator

mumoshu commented Jul 26, 2021

FYI, we already have a lot of unit tests like https://github.com/roboll/helmfile/blob/master/pkg/app/app_test.go#L2749 that are for avoiding regressions like this. At least unit tests are passing. Especially the smoke test covers multiple releases with a complex DAG so needs should work. Apparently there're still edge-cases?

@dudicoco
Copy link
Contributor Author

@mumoshu if it's a PR for fixing the bug, I don't think i'll be able to find and fix the problem within the code :)
If it's for reverting the include-needs feature, I could give it a shot.

Regarding the smoke test, the use case here is very simple and straightforward use of needs so I don't believe it's an edge-case, I've noticed that the smoke test uses releases without namespace prefixed, perhaps this is why they are passing?

@mumoshu
Copy link
Collaborator

mumoshu commented Jan 7, 2022

I recently verified needs to work while verifying another issue #1959. For #1959, the problem was that the need is silently ignored if you had any typo in namespace or name of the needed release. So I added an explicit error in that case, in #2026.

Everyone, would you mind sharing a single helmfile.yaml that needs doesn't work with?
It's already a half year since this issue has been created but I didn't receive any single helmfile.yaml for reproduction, and I couldn't reproduce the issue with my own examples.

The first example provided by @dudicoco in this issue lacked namespace in the needed release but he says it's a kind of a base template that is source of the final helmfile.yaml 🤔 #1881 (comment)

@dudicoco
Copy link
Contributor Author

dudicoco commented Jan 9, 2022

Hi @mumoshu, here's a more complete helmfile.yaml for reproduction:

- name: amazing-app-test-1
  chart: my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - my-context/devops/amazing-app-test-3
  values: []
- name: amazing-app-test-2
  chart: my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - my-context/devops/amazing-app-test-1
  values: []
- name: amazing-app-test-3
  chart: my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs: []
  values: []

Let me know if anything else is needed.

@mumoshu
Copy link
Collaborator

mumoshu commented Jan 10, 2022

@dudicoco Thanks! FWIW, I couldn't reproduce the problem with helmfile 0.139.9

Upgrading release=amazing-app-test-3, chart=my-repo/amazing-app
Release "amazing-app-test-3" does not exist. Installing it now.
NAME: amazing-app-test-3
LAST DEPLOYED: Mon Jan 10 06:07:46 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-3" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-3$
amazing-app-test-3      devops          1               2022-01-10 06:07:46.038450005 +0000 UTC  deployed        amazing-app-0.1.01.16.0     

Upgrading release=amazing-app-test-1, chart=my-repo/amazing-app
Release "amazing-app-test-1" does not exist. Installing it now.
NAME: amazing-app-test-1
LAST DEPLOYED: Mon Jan 10 06:07:46 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-1" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-1$
amazing-app-test-1      devops          1               2022-01-10 06:07:46.682626195 +0000 UTC  deployed        amazing-app-0.1.01.16.0     

Upgrading release=amazing-app-test-2, chart=my-repo/amazing-app
Release "amazing-app-test-2" does not exist. Installing it now.
NAME: amazing-app-test-2
LAST DEPLOYED: Mon Jan 10 06:07:47 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-2" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-2$
amazing-app-test-2      devops          1               2022-01-10 06:07:47.335290788 +0000 UTC  deployed        amazing-app-0.1.01.16.0     


UPDATED RELEASES:
NAME                 CHART                 VERSION
amazing-app-test-3   my-repo/amazing-app     0.1.0
amazing-app-test-1   my-repo/amazing-app     0.1.0
amazing-app-test-2   my-repo/amazing-app     0.1.0

@dudicoco
Copy link
Contributor Author

@mumoshu what command did you use to install the releases?

I've used helmfile -e my-environment -l name=amazing-app-test-1 -l name=amazing-app-test-2 -l name=amazing-app-test-3 apply

@mumoshu
Copy link
Collaborator

mumoshu commented Jan 10, 2022

@dudicoco I just ran plain helmfile apply. I also ran it with -l like the below but it seems to work:

./helmfile-0.139.9 -f helmfile.1881.yaml -l name=amazing-app-test-1 -l name=amazing-app-test-2 -l name=amazing-app-test-3 apply
Upgrading release=amazing-app-test-3, chart=my-repo/amazing-app
Release "amazing-app-test-3" does not exist. Installing it now.
NAME: amazing-app-test-3
LAST DEPLOYED: Mon Jan 10 12:37:36 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-3" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-3$
amazing-app-test-3      devops          1               2022-01-10 12:37:36.013775617 +0000 UTC  deployed        amazing-app-0.1.01.16.0     

Upgrading release=amazing-app-test-1, chart=my-repo/amazing-app
Release "amazing-app-test-1" does not exist. Installing it now.
NAME: amazing-app-test-1
LAST DEPLOYED: Mon Jan 10 12:37:36 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-1" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-1$
amazing-app-test-1      devops          1               2022-01-10 12:37:36.658368508 +0000 UTC  deployed        amazing-app-0.1.01.16.0     

Upgrading release=amazing-app-test-2, chart=my-repo/amazing-app
Release "amazing-app-test-2" does not exist. Installing it now.
NAME: amazing-app-test-2
LAST DEPLOYED: Mon Jan 10 12:37:37 2022
NAMESPACE: devops
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace devops -l "app.kubernetes.io/name=amazing-app,app.kubernetes.io/instance=amazing-app-test-2" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace devops $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace devops port-forward $POD_NAME 8080:$CONTAINER_PORT

Listing releases matching ^amazing-app-test-2$
amazing-app-test-2      devops          1               2022-01-10 12:37:37.278351885 +0000 UTC  deployed        amazing-app-0.1.01.16.0     


UPDATED RELEASES:
NAME                 CHART                 VERSION
amazing-app-test-3   my-repo/amazing-app     0.1.0
amazing-app-test-1   my-repo/amazing-app     0.1.0
amazing-app-test-2   my-repo/amazing-app     0.1.0

@dudicoco
Copy link
Contributor Author

@mumoshu i've just tried running it many times with different configurations in my helmfile.yaml and the issue did not go away.

Could there be a difference between our helmfile versions? Is there any other information I can provide you with? perhaps debug logs?

@mumoshu
Copy link
Collaborator

mumoshu commented Jan 10, 2022

@dudicoco Would you mind sharing me a github repo with a complete example of a helmfile project that can reproduce the issue?

Also which OS and CPU arch are you using?

I use helmfile 0.139.9 on Ubuntu 20.04.

I don't actually have access to my-repo/amazing-app in your example so my example is like the below. ./my-repo/amazing-app points to a ad-hoc that created by running mkdir my-repo && helm create my-repo/amazing-app

releases:
- name: amazing-app-test-1
  chart: ./my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - my-context/devops/amazing-app-test-3
  values: []
- name: amazing-app-test-2
  chart: ./my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs:
  - my-context/devops/amazing-app-test-1
  values: []
- name: amazing-app-test-3
  chart: ./my-repo/amazing-app
  kubeContext: my-context
  namespace: devops
  version: 2.0.14
  installed: true
  timeout: 300
  needs: []
  values: []

@dudicoco
Copy link
Contributor Author

@mumoshu please try it with chart bitnami/nginx version 9.7.1: https://artifacthub.io/packages/helm/bitnami/nginx

I'm using helmfile version v0.139.9 on darwin x86_64.

@dudicoco
Copy link
Contributor Author

Hi @mumoshu, how can we track this issue in the new repo? Should I just reopen it there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants