Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k8s lookup return a dictionary #9

Closed
jobcespedes opened this issue Apr 4, 2021 · 9 comments · Fixed by #117
Closed

k8s lookup return a dictionary #9

jobcespedes opened this issue Apr 4, 2021 · 9 comments · Fixed by #117
Assignees
Labels
type/bug Something isn't working

Comments

@jobcespedes
Copy link

SUMMARY

Using the k8s lookup, it returns a dictionary. I was expecting a list

ISSUE TYPE
  • Bug Report
COMPONENT NAME

k8s lookup

ANSIBLE VERSION
ansible 2.9.16
  config file = None
  configured module search path = ['/var/home/job/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /var/home/job/.local/lib/python3.8/site-packages/ansible
  executable location = /var/home/job/.local/bin/ansible
  python version = 3.8.7 (default, Dec 22 2020, 00:00:00) [GCC 10.2.1 20201125 (Red Hat 10.2.1-9)]
CONFIGURATION

OS / ENVIRONMENT

Fedora 32

STEPS TO REPRODUCE
  - vars:
      namespace: osdk-pr-0-0
      php_fpm_appname: m4e-sample-php-fpm
    debug:
      msg: "{{ lookup('k8s', namespace=namespace, kind='pod', label_selector='app=' +
        php_fpm_appname, field_selector='status.phase=Running') }}"
EXPECTED RESULTS

A list; even though, there is only one item

ACTUAL RESULTS

A dict. There is only one item

TASK [debug] **************************************************************************************************************************
task path: /var/home/job/mydata/myrepos/m4e-operator/.idea/debug.yml:15
ok: [localhost] => {
    "msg": {
        "apiVersion": "v1",
        "kind": "Pod",
        "metadata": {
            "creationTimestamp": "2021-04-04T03:59:35Z",
            "generateName": "m4e-sample-php-fpm-deploy-74c9847c79-",
            "labels": {
                "app": "m4e-sample-php-fpm",
                "app.kubernetes.io/component": "php-fpm",
                "app.kubernetes.io/instance": "m4e-sample",
                "app.kubernetes.io/managed-by": "ansible",
                "app.kubernetes.io/name": "m4e-sample-php-fpm",
                "app.kubernetes.io/part-of": "moodle",
                "app.kubernetes.io/version": "v1alpha1",
                "pod-template-hash": "74c9847c79"
            },
            "managedFields": [
                {
                    "apiVersion": "v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:metadata": {
                            "f:generateName": {},
                            "f:labels": {
                                ".": {},
                                "f:app": {},
                                "f:app.kubernetes.io/component": {},
                                "f:app.kubernetes.io/instance": {},
                                "f:app.kubernetes.io/managed-by": {},
                                "f:app.kubernetes.io/name": {},
                                "f:app.kubernetes.io/part-of": {},
                                "f:app.kubernetes.io/version": {},
                                "f:pod-template-hash": {}
                            },
                            "f:ownerReferences": {
                                ".": {},
                                "k:{\"uid\":\"2233a2d7-8cfd-4c5b-bac0-87194a702bc6\"}": {
                                    ".": {},
                                    "f:apiVersion": {},
                                    "f:blockOwnerDeletion": {},
                                    "f:controller": {},
                                    "f:kind": {},
                                    "f:name": {},
                                    "f:uid": {}
                                }
                            }
                        },
                        "f:spec": {
                            "f:containers": {
                                "k:{\"name\":\"m4e-sample-php-fpm\"}": {
                                    ".": {},
                                    "f:args": {},
                                    "f:env": {
                                        ".": {},
                                        "k:{\"name\":\"MOODLE_CONFIG_DIR\"}": {
                                            ".": {},
                                            "f:name": {},
                                            "f:value": {}
                                        },
                                        "k:{\"name\":\"PHP_FPM_LISTEN_ALLOWED_CLIENTS\"}": {
                                            ".": {},
                                            "f:name": {},
                                            "f:value": {}
                                        },
                                        "k:{\"name\":\"PHP_FPM_PROCESS_CONTROL_TIMEOUT\"}": {
                                            ".": {},
                                            "f:name": {},
                                            "f:value": {}
                                        }
                                    },
                                    "f:image": {},
                                    "f:imagePullPolicy": {},
                                    "f:livenessProbe": {
                                        ".": {},
                                        "f:exec": {
                                            ".": {},
                                            "f:command": {}
                                        },
                                        "f:failureThreshold": {},
                                        "f:initialDelaySeconds": {},
                                        "f:periodSeconds": {},
                                        "f:successThreshold": {},
                                        "f:timeoutSeconds": {}
                                    },
                                    "f:name": {},
                                    "f:ports": {
                                        ".": {},
                                        "k:{\"containerPort\":9000,\"protocol\":\"TCP\"}": {
                                            ".": {},
                                            "f:containerPort": {},
                                            "f:protocol": {}
                                        }
                                    },
                                    "f:readinessProbe": {
                                        ".": {},
                                        "f:exec": {
                                            ".": {},
                                            "f:command": {}
                                        },
                                        "f:failureThreshold": {},
                                        "f:initialDelaySeconds": {},
                                        "f:periodSeconds": {},
                                        "f:successThreshold": {},
                                        "f:timeoutSeconds": {}
                                    },
                                    "f:resources": {
                                        ".": {},
                                        "f:limits": {
                                            ".": {},
                                            "f:cpu": {},
                                            "f:memory": {}
                                        },
                                        "f:requests": {
                                            ".": {},
                                            "f:cpu": {},
                                            "f:memory": {}
                                        }
                                    },
                                    "f:terminationMessagePath": {},
                                    "f:terminationMessagePolicy": {},
                                    "f:volumeMounts": {
                                        ".": {},
                                        "k:{\"mountPath\":\"/config\"}": {
                                            ".": {},
                                            "f:mountPath": {},
                                            "f:name": {},
                                            "f:readOnly": {}
                                        },
                                        "k:{\"mountPath\":\"/var/moodledata\"}": {
                                            ".": {},
                                            "f:mountPath": {},
                                            "f:name": {}
                                        }
                                    }
                                }
                            },
                            "f:dnsPolicy": {},
                            "f:enableServiceLinks": {},
                            "f:restartPolicy": {},
                            "f:schedulerName": {},
                            "f:securityContext": {
                                ".": {},
                                "f:fsGroup": {},
                                "f:runAsUser": {}
                            },
                            "f:terminationGracePeriodSeconds": {},
                            "f:volumes": {
                                ".": {},
                                "k:{\"name\":\"config-php\"}": {
                                    ".": {},
                                    "f:name": {},
                                    "f:secret": {
                                        ".": {},
                                        "f:defaultMode": {},
                                        "f:items": {},
                                        "f:secretName": {}
                                    }
                                },
                                "k:{\"name\":\"moodledata\"}": {
                                    ".": {},
                                    "f:name": {},
                                    "f:persistentVolumeClaim": {
                                        ".": {},
                                        "f:claimName": {}
                                    }
                                }
                            }
                        }
                    },
                    "manager": "kube-controller-manager",
                    "operation": "Update",
                    "time": "2021-04-04T03:59:35Z"
                },
                {
                    "apiVersion": "v1",
                    "fieldsType": "FieldsV1",
                    "fieldsV1": {
                        "f:status": {
                            "f:conditions": {
                                "k:{\"type\":\"ContainersReady\"}": {
                                    ".": {},
                                    "f:lastProbeTime": {},
                                    "f:lastTransitionTime": {},
                                    "f:status": {},
                                    "f:type": {}
                                },
                                "k:{\"type\":\"Initialized\"}": {
                                    ".": {},
                                    "f:lastProbeTime": {},
                                    "f:lastTransitionTime": {},
                                    "f:status": {},
                                    "f:type": {}
                                },
                                "k:{\"type\":\"Ready\"}": {
                                    ".": {},
                                    "f:lastProbeTime": {},
                                    "f:lastTransitionTime": {},
                                    "f:status": {},
                                    "f:type": {}
                                }
                            },
                            "f:containerStatuses": {},
                            "f:hostIP": {},
                            "f:phase": {},
                            "f:podIP": {},
                            "f:podIPs": {
                                ".": {},
                                "k:{\"ip\":\"10.88.136.128\"}": {
                                    ".": {},
                                    "f:ip": {}
                                }
                            },
                            "f:startTime": {}
                        }
                    },
                    "manager": "kubelet",
                    "operation": "Update",
                    "time": "2021-04-04T04:00:04Z"
                }
            ],
            "name": "m4e-sample-php-fpm-deploy-74c9847c79-gpb6b",
            "namespace": "osdk-pr-0-0",
            "ownerReferences": [
                {
                    "apiVersion": "apps/v1",
                    "blockOwnerDeletion": true,
                    "controller": true,
                    "kind": "ReplicaSet",
                    "name": "m4e-sample-php-fpm-deploy-74c9847c79",
                    "uid": "2233a2d7-8cfd-4c5b-bac0-87194a702bc6"
                }
            ],
            "resourceVersion": "68158643",
            "uid": "228a69ba-28fb-42f3-973c-b3c8a261dd6f"
        },
        "spec": {
            "containers": [
                {
                    "args": [
                        "php-fpm"
                    ],
                    "env": [
                        {
                            "name": "PHP_FPM_LISTEN_ALLOWED_CLIENTS",
                            "value": "any"
                        },
                        {
                            "name": "PHP_FPM_PROCESS_CONTROL_TIMEOUT",
                            "value": "20"
                        },
                        {
                            "name": "MOODLE_CONFIG_DIR",
                            "value": "/config"
                        }
                    ],
                    "image": "quay.io/krestomatio/moodle_web",
                    "imagePullPolicy": "Always",
                    "livenessProbe": {
                        "exec": {
                            "command": [
                                "/usr/libexec/check-container-php",
                                "-t",
                                "-l"
                            ]
                        },
                        "failureThreshold": 3,
                        "initialDelaySeconds": 5,
                        "periodSeconds": 10,
                        "successThreshold": 1,
                        "timeoutSeconds": 3
                    },
                    "name": "m4e-sample-php-fpm",
                    "ports": [
                        {
                            "containerPort": 9000,
                            "protocol": "TCP"
                        }
                    ],
                    "readinessProbe": {
                        "exec": {
                            "command": [
                                "/usr/libexec/check-container-php",
                                "-t",
                                "-r"
                            ]
                        },
                        "failureThreshold": 6,
                        "initialDelaySeconds": 5,
                        "periodSeconds": 30,
                        "successThreshold": 1,
                        "timeoutSeconds": 3
                    },
                    "resources": {
                        "limits": {
                            "cpu": "1",
                            "memory": "1Gi"
                        },
                        "requests": {
                            "cpu": "150m",
                            "memory": "256Mi"
                        }
                    },
                    "terminationMessagePath": "/dev/termination-log",
                    "terminationMessagePolicy": "File",
                    "volumeMounts": [
                        {
                            "mountPath": "/var/moodledata",
                            "name": "moodledata"
                        },
                        {
                            "mountPath": "/config",
                            "name": "config-php",
                            "readOnly": true
                        },
                        {
                            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                            "name": "default-token-k2qkg",
                            "readOnly": true
                        }
                    ]
                }
            ],
            "dnsPolicy": "ClusterFirst",
            "enableServiceLinks": true,
            "nodeName": "minikube",
            "preemptionPolicy": "PreemptLowerPriority",
            "priority": 0,
            "restartPolicy": "Always",
            "schedulerName": "default-scheduler",
            "securityContext": {
                "fsGroup": 48,
                "runAsUser": 48
            },
            "serviceAccount": "default",
            "serviceAccountName": "default",
            "terminationGracePeriodSeconds": 30,
            "tolerations": [
                {
                    "effect": "NoExecute",
                    "key": "node.kubernetes.io/not-ready",
                    "operator": "Exists",
                    "tolerationSeconds": 300
                },
                {
                    "effect": "NoExecute",
                    "key": "node.kubernetes.io/unreachable",
                    "operator": "Exists",
                    "tolerationSeconds": 300
                }
            ],
            "volumes": [
                {
                    "name": "moodledata",
                    "persistentVolumeClaim": {
                        "claimName": "m4e-sample-moodle-data"
                    }
                },
                {
                    "name": "config-php",
                    "secret": {
                        "defaultMode": 420,
                        "items": [
                            {
                                "key": "config.php",
                                "path": "config.php"
                            }
                        ],
                        "secretName": "m4e-sample-moodle-secret"
                    }
                },
                {
                    "name": "default-token-k2qkg",
                    "secret": {
                        "defaultMode": 420,
                        "secretName": "default-token-k2qkg"
                    }
                }
            ]
        },
        "status": {
            "conditions": [
                {
                    "lastProbeTime": null,
                    "lastTransitionTime": "2021-04-04T03:59:36Z",
                    "status": "True",
                    "type": "Initialized"
                },
                {
                    "lastProbeTime": null,
                    "lastTransitionTime": "2021-04-04T04:00:04Z",
                    "status": "True",
                    "type": "Ready"
                },
                {
                    "lastProbeTime": null,
                    "lastTransitionTime": "2021-04-04T04:00:04Z",
                    "status": "True",
                    "type": "ContainersReady"
                },
                {
                    "lastProbeTime": null,
                    "lastTransitionTime": "2021-04-04T03:59:36Z",
                    "status": "True",
                    "type": "PodScheduled"
                }
            ],
            "containerStatuses": [
                {
                    "containerID": "docker://b77ee173496abb37a0092d2869cf1efb59c2af43adbf5ecab1d256e0108e9836",
                    "image": "quay.io/krestomatio/moodle_web:latest",
                    "imageID": "docker-pullable://quay.io/krestomatio/moodle_web@sha256:48d069f0bc5301a547966ea20050d1b3fcfeb8dce871c65230cf8d6d62eb867c",
                    "lastState": {},
                    "name": "m4e-sample-php-fpm",
                    "ready": true,
                    "restartCount": 0,
                    "started": true,
                    "state": {
                        "running": {
                            "startedAt": "2021-04-04T03:59:38Z"
                        }
                    }
                }
            ],
            "hostIP": "192.168.39.222",
            "phase": "Running",
            "podIP": "10.88.136.128",
            "podIPs": [
                {
                    "ip": "10.88.136.128"
                }
            ],
            "qosClass": "Burstable",
            "startTime": "2021-04-04T03:59:36Z"
        }
    }
}
META: ran handlers
META: ran handlers
@Akasurde
Copy link
Member

Akasurde commented Apr 5, 2021

@jobcespedes Thanks for reporting this issue. I can confirm when there is the only an item returned by API, lookup plugin returns a dictionary.

diff --git a/plugins/lookup/k8s.py b/plugins/lookup/k8s.py
index fc4558c..a52f698 100644
--- a/plugins/lookup/k8s.py
+++ b/plugins/lookup/k8s.py
@@ -278,7 +278,11 @@ class KubernetesLookup(K8sAnsibleMixin):
         if self.name:
             return [k8s_obj.to_dict()]

-        return k8s_obj.to_dict().get('items')
+        items = k8s_obj.to_dict().get('items')
+
+        if len(items) == 1:
+            return [items]
+        return items

I will check if there are any other cases where it returns a dictionary.

@gravesm gravesm transferred this issue from ansible-collections/community.kubernetes Apr 8, 2021
@goneri goneri added the type/bug Something isn't working label Apr 9, 2021
@goneri
Copy link
Member

goneri commented Apr 9, 2021

@Akasurde the fix will introduce a breaking change from a user perspective. Maybe we can try to target 2.0.0 because of that.

@Akasurde
Copy link
Member

@jobcespedes Could you please confirm if this fix resolves your issue and let me know? Thanks.

@Akasurde
Copy link
Member

@Akasurde the fix will introduce a breaking change from a user perspective. Maybe we can try to target 2.0.0 because of that.

@goneri This is a bug in the behavior, so it is not a breaking change.

@jobcespedes
Copy link
Author

@jobcespedes Could you please confirm if this fix resolves your issue and let me know? Thanks.

Confirm it returns a list with one item

Akasurde added a commit to Akasurde/kubernetes.core that referenced this issue Jun 1, 2021
Always return list from k8s lookup plugin

Fixes: ansible-collections#9

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Akasurde added a commit to Akasurde/kubernetes.core that referenced this issue Jun 1, 2021
Always return list from k8s lookup plugin

Fixes: ansible-collections#9

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Akasurde added a commit to Akasurde/kubernetes.core that referenced this issue Jun 1, 2021
Always return list from k8s lookup plugin

Fixes: ansible-collections#9

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Akasurde added a commit to Akasurde/kubernetes.core that referenced this issue Jun 1, 2021
Always return list from k8s lookup plugin

Fixes: ansible-collections#9

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Akasurde added a commit that referenced this issue Jun 4, 2021
Always return list from k8s lookup plugin

Fixes: #9

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
@jmazzitelli
Copy link
Contributor

I reported some issues surrounding this - see: #147

FWIW: I think this issue was incorrect - it said in the description: "Using the k8s lookup, it returns a dictionary. I was expecting a list." You should not expect a list from lookup, you should expect a string. You should expect a list from query. Only if you pass "wantlist=True" should you expect lookup to return a list.

At least, that is how I read the ansible docs here: https://docs.ansible.com/ansible/latest/plugins/lookup.html#forcing-lookups-to-return-lists-query-and-wantlist-true

Notice it says,

In Ansible 2.5, a new Jinja2 function called query was added for invoking lookup plugins.
The difference between lookup and query is largely that query will always return a list.
The default behavior of lookup is to return a string of comma separated values.
lookup can be explicitly configured to return a list using wantlist=True.

Notice it says "default behavior of lookup is to return a string of comma separated values." - thus, lookup should not return a list unless you specify "wantlist=True".

The reason for me posting this comment here is - I would like confirmation if my analysis above is correct or if it is wrong in some way.

@jobcespedes
Copy link
Author

A list is expected in my interpretation from the k8s lookup docs: https://docs.ansible.com/ansible/latest/collections/community/kubernetes/k8s_lookup.html#return-_list

@jmazzitelli I am probably not the best person to confirm whether you are correct or not. However, reading what you shared, it doesn't tell me a lookup plugin should not return a list.

¿Do you think k8s lookup should return a representation of k8s objects in a string of comma separated values?

@jmazzitelli
Copy link
Contributor

¿Do you think k8s lookup should return a representation of k8s objects in a string of comma separated values?

Based on that doc you referenced: https://docs.ansible.com/ansible/latest/collections/community/kubernetes/k8s_lookup.html#return-_list it does seem to say it returns a list always. But even today (even with this issue fixed) that is not true. It does not return a list when you use resource_name as opposed to label_selector.

So, I dunno - the documentation conflicts here and the implementation is confused.

Based on this page: https://docs.ansible.com/ansible/latest/plugins/lookup.html#forcing-lookups-to-return-lists-query-and-wantlist-true the lookup should not be returning a list by default (without wantlist=True) - if you want a list, use query (again, based on those docs). At the very least, I would expect lookup to return a single resource when a single resource results (I shouldn't have to get the [0] index when I am expecting a single result, as in the case when using resource_name). If I wanted to do that, I would use query instead. Indeed, this is the exact behavior of lookup when using resource_name (as opposed to label_selector) - see my results here that show this.

So the behavior of k8s lookup is STILL not returning a list always (e.g. in the case you use resource_name).

It's a big jumble.

...shrug... we'd need someone on the ansible collections team to respond. Something needs to change - the docs, the implementation. I dunno :)

@Akasurde
Copy link
Member

@jobcespedes @jmazzitelli I will work on this and confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants