Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) #110

nolar · 2019-06-14T09:29:11Z

Issue : closes #15, closes #84, closes #137, related #97 (docs).

Description

Switch the internal K8s client library from the official library kubernetes to pykube-ng.

This simplifies handling of arbitrary resource types, such as pods, jobs, etc — for any handlers: both regular handlers (@kopf.on.create/update/delete/resume), and spy-handles (@kopf.on.event).

The previous Kubernetes client library is supported optionally: if present, it will be auto-logged in, to keep compatibility with the previous behaviour. Otherwise, it is ignored. And it is not used anywhere. It will be removed from 1.0 release.

Types of Changes

New feature (non-breaking change which adds functionality)
Refactor/improvements

PLEASE NOTE: The K8s client library used inside of Kopf is Kopf's internal implementation detail. Only the native data structures (dicts, lists, etc) are exposed from the framework to the operator and accepted back. The operators MUST NOT rely on how Kopf talks to the K8s APIs. It can be requests/requests-async in the future (just as an example). The only dependency is auto-login — to be removed with #59 (the operators must explicitly authenticate themselves; now they can do this on the module-level code only).

Todos

Add and examples, revert examples 01/02 to what they were initially.
Extend the docs to point that arbitrary resources are supported.
Ensure it works in all cases with all features and nothing is broken (a real test-drive for 1-2 weeks).

Review

List of tasks the reviewer must do to review the PR

Tests
Documentation

zincr · 2019-06-14T09:29:24Z

🤖 zincr found 0 problems , 2 warnings

ℹ️ Large Commits
ℹ️ Dependency Licensing
✅ Approvals
✅ Specification

Details on how to resolve are provided below

Large Commits

Checks all commits for large additions to a single file. Large commits should be reviewed more carefully for potential copyright and licensing issues

This file contains a substantial change, please review to determine if the change comes from an external source and if there are any copyright or licensing issues to be aware of

ℹ️ kopf/clients/auth.py had +100 lines of code changed in a single commit
Please review this file to determine the source of this change
ℹ️ kopf/clients/fetching.py had +100 lines of code changed in a single commit
Please review this file to determine the source of this change
ℹ️ tests/cli/test_login.py had +100 lines of code changed in a single commit
Please review this file to determine the source of this change
ℹ️ tests/conftest.py had +100 lines of code changed in a single commit
Please review this file to determine the source of this change
ℹ️ tests/k8s/test_patching.py had +100 lines of code changed in a single commit
Please review this file to determine the source of this change

Dependency Licensing

All dependencies specified in package manager files must be reviewed, banned dependency licenses will block the merge, all new dependencies introduced in this pull request will give a warning, but not block the merge

Please ensure that only dependencies with licenses compatible with the license of this project is included in the pull request.

ℹ️ Detected pykube-ng as a dependency in examples/requirements.txt, licensed under: Apache-2.0
please review it and confirm you wish to introduce this to the codebase

angelbarrera92 · 2019-06-15T08:46:46Z

I tried this PR locally on K3s and i was having connectivity problems.
For those who want to try it on k3s locally (fedora), please go to the following k3s issue: k3s-io/k3s#24

Thanks @nolar for your support!

hjacobs · 2019-06-15T10:54:49Z

kopf/k8s/classes.py

+from kopf.k8s import config
+
+
+# TODO: this mixin has to be migrated into pykube-ng itself.


feel free to open a PR for pykube-ng 😄

I will for sure. First of all, let's see how it works in this mode. And second:

With the official client, there was an issue when the switch the Content-Type from merge-patch format (dicts of fields) to JSON-patch format (lists of ops) — see #134 and kubernetes-client/python#866.

For Kopf, merge-patch is needed and is sufficient. But for a generic k8s client, both formats should be supported.

hjacobs · 2019-06-15T10:55:18Z

kopf/k8s/classes.py

+    kind = "CustomResourceDefinition"
+
+
+def _make_cls(resource) -> Type[APIObject]:


I guess this can be part of pykube-ng, too

@hjacobs In theory, there is already an implementation in pykube.object_factory(), and it is an officially documented way of creating the classes.

But I had issues with it. First, because of the self-made mixins with the patch() method (now solved). Second, because it requires kind as a search criterion, and Kopf uses the canonical plural names from the name field (as in the K8s API URLs, and as in the original Kubernetes client library).

The automatic implicit mapping kubectl-style (i.e. any form is accepted, including the short-names) is planned in #57, and it will be more convenient for the users, but it is not yet there.

It would be nice if pukube-ng could search by any name field (kind, plural, singular, all shortnames) — same as kubectl does. But it can be done as a separate PR in Kopf.

hjacobs · 2019-06-15T10:56:27Z

kopf/k8s/config.py

+
+def login_pykube():
+    global _pykube_cfg
+    try:


I guess we can add some "auto.." method for pykube-ng itself to not repeat the logic everywhere (I do the same as you in my projects: first try "in-cluster" then try kubeconfig file)

nolar · 2019-07-08T13:37:13Z

All pre-requisites are now in master. This branch is rebased on top of master, with few fixes here & there. The implementation is mainly the same.

Ready for review — to be released as a pre-release version (presumably 0.19rc1), and to be tested in our own apps before going to the public release.

hjacobs · 2019-07-08T15:02:16Z

@nolar I created a PR to move reasonable changes upstream to pykube-ng: hjacobs/pykube#29

examples/10-builtins/README.md

examples/requirements.txt

kopf/clients/auth.py

thilp · 2019-07-08T15:15:41Z

kopf/clients/auth.py

+            raise AccessError("Cannot authenticate to the Kubernetes API. "
+                              "Please login or configure the tokens.")
+        else:
+            raise


I was surprised to get a 403 from there the other day. I eventually discovered it was because my operator's service account was missing the following role:

nonResourceURLs: ["/"] verbs: ["get"]

(This was not a problem when Kopf was using the kubernetes client.)
Do you think it would be worth catching and mentioning here (not necessarily in this PR)?

Good catch, I think requesting / might be tricky to not forget in RBAC setups

Good point. Maybe, there should be no verification at all. It should fail on the actual operations attempted.

Looking wider, it can be so that authentication succeeds, but the authorization fails on specific resources or specific REST operations of those resources, such as PATCH (because of RBAC). The operator should perform a full self-check on startup. It will only cover the permissions needed by the framework (watching-patching), but will not cover the permissions needed by the operator's logic (e.g. creating pods/jobs).

See also: #49 for such self-checks (against a RBAC yaml, not against the real cluster, but it can be extended).

Meanwhile, I will make it here to ignore all errors except for HTTP 401 and TCP/HTTPS connection errors — since the purpose is to verify authentication, not authorization.

Done in 2118b8e. Now, only connection errors and 401 do raise (and few tests added). All other errors are ignored.

kopf/clients/auth.py

kopf/clients/classes.py

kopf/logging/logging.py

tools/kubernetes-client.sh

hjacobs · 2019-07-08T15:47:28Z

kopf/clients/classes.py

+
+
+# TODO: this mixin has to be migrated into pykube-ng itself.
+class APIObject(pykube.objects.APIObject):


I created hjacobs/pykube#29 to have this upstream

@hjacobs I've switched to pykube-ng>=0.25. So, the patching & CRD hacks are gone (including these APIObject/NamespacesAPIObject classes). See f105def

hjacobs · 2019-07-08T15:47:35Z

kopf/clients/classes.py

+    pass
+
+
+class CustomResourceDefinition(APIObject):


I created hjacobs/pykube#29 to have this upstream

Released in https://github.com/hjacobs/pykube/releases/tag/0.24

codecov-io · 2019-07-09T10:05:00Z

Codecov Report

Merging #110 into master will decrease coverage by 0.15%.
The diff coverage is 85%.

@@            Coverage Diff             @@
##           master     #110      +/-   ##
==========================================
- Coverage   83.81%   83.65%   -0.16%     
==========================================
  Files          29       30       +1     
  Lines        1433     1542     +109     
  Branches      211      226      +15     
==========================================
+ Hits         1201     1290      +89     
- Misses        194      208      +14     
- Partials       38       44       +6

Flag	Coverage Δ
#unit	`83.65% <85%> (-0.16%)`	⬇️

Impacted Files	Coverage Δ
kopf/engines/peering.py	`28.34% <0%> (+0.56%)`	⬆️
kopf/config.py	`91.8% <100%> (+0.89%)`	⬆️
kopf/reactor/registries.py	`90% <100%> (+0.79%)`	⬆️
kopf/clients/events.py	`84.84% <100%> (+0.47%)`	⬆️
kopf/cli.py	`87.5% <100%> (+1.53%)`	⬆️
kopf/clients/patching.py	`100% <100%> (ø)`	⬆️
kopf/clients/watching.py	`91.66% <66.66%> (-4.17%)`	⬇️
kopf/clients/classes.py	`71.42% <71.42%> (ø)`
kopf/clients/fetching.py	`82.97% <78.37%> (-17.03%)`	⬇️
kopf/clients/auth.py	`86.84% <85%> (+3.5%)`	⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 755d36c...f105def. Read the comment docs.

nolar · 2019-07-09T11:07:09Z

@thilp @hjacobs All issues are addressed (I think). Can be re-reviewed. Please, comment/highlight again if I missed something.

thilp

Just a potential simplification, but everything looks great.

thilp · 2019-07-09T12:29:40Z

kopf/clients/auth.py

    """
    try:
        api = get_pykube_api()
        rsp = api.get(version="", base="/")
        rsp.raise_for_status()
+        api.raise_for_status(rsp)  # replaces requests's HTTPError with its own.


I think the rsp.raise_for_status() can be removed, because api.raise_for_status(rsp) already does it and you are already catching everything.

Basically if the first line raises, the second would as well (but will not be run because the first raised), and if the first line doesn't raise, then the second won't either, so nothing is gained by having both.

nolar · 2019-07-09T13:18:38Z

Released as kopf==0.19rc1.

Note: it is a pre-release, so it requires either pip install --pre kopf, or explicit version pinning: pip install kopf==0.19.rc1.

nolar added the enhancement New feature or request label Jun 14, 2019

nolar added this to the 1.0 milestone Jun 14, 2019

nolar requested a review from samurang87 as a code owner June 14, 2019 09:29

nolar force-pushed the pykube branch from 726bff9 to 2ed676b Compare June 14, 2019 20:22

hjacobs reviewed Jun 15, 2019

View reviewed changes

nolar force-pushed the pykube branch from 2ed676b to 125c5d0 Compare June 18, 2019 10:13

nolar mentioned this pull request Jun 19, 2019

Update the EphemeralVolumeClaim story-telling #97

Merged

7 tasks

Sergey Vasilyev added 10 commits July 8, 2019 15:13

Switch to pykube-ng, keep kubernetes client optional

e53a598

Test that login works when kubernetes library is not installed

6331c1a

Test with kubernetes client library & its dependencies not pre-installed

490a4d3

Pass the list's resource version for watching init state

05a3088

Simulate the missing resource kind identifiers in the initial listing

41e1ff6

Combine k8s login & access check into one routine, disabled by default

1a7faef

Add an example for built-in pods controller

0f021c7

Verify the importing machinery restored properly after import hacks

d225cfa

Make all client libraries optional for mocks, but do not skip the tests

27da45c

Skip e2e tests if they require kubernetes client when it is absent

1e2b806

nolar force-pushed the pykube branch from 125c5d0 to 1e2b806 Compare July 8, 2019 13:32

hjacobs mentioned this pull request Jul 8, 2019

Kopf changes: add CRD, patch method hjacobs/pykube#29

Merged

thilp reviewed Jul 8, 2019

View reviewed changes

hjacobs reviewed Jul 8, 2019

View reviewed changes

nolar mentioned this pull request Jul 9, 2019

Can this framework be used to monitor core resources like namespaces? #137

Closed

Remove a note for an already solved issue

44602c1

Sergey Vasilyev added 3 commits July 9, 2019 11:54

Raise only on connection and authentication errors, ignore permissions

2118b8e

Use the default timeout of pykube if not passed to get_pykube_api()

0db1078

Specify which CRD is absent in an exception

495744b

Sergey Vasilyev added 2 commits July 9, 2019 12:34

Remove a file that accidentally sneaked in

f75fa7a

Switch to pykube-ng version with patching & CRDs added

f105def

thilp approved these changes Jul 9, 2019

View reviewed changes

nolar merged commit 700c7d1 into zalando-incubator:master Jul 9, 2019

nolar deleted the pykube branch July 9, 2019 13:18

nolar mentioned this pull request Jul 11, 2019

Disable socket timeouts for internal request, especially for watching #144

Merged

nolar mentioned this pull request Oct 1, 2019

Remove an abstraction leak of a Response object — for type-hinting #197

Merged

kopf-archiver bot mentioned this pull request Aug 19, 2020

[PR] Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) nolar/kopf#110

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) #110

Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) #110

nolar commented Jun 14, 2019 •

edited

Loading

zincr bot commented Jun 14, 2019 •

edited

Loading

angelbarrera92 commented Jun 15, 2019

hjacobs Jun 15, 2019

nolar Jul 8, 2019

hjacobs Jun 15, 2019

nolar Jul 9, 2019 •

edited

Loading

hjacobs Jun 15, 2019

nolar commented Jul 8, 2019

hjacobs commented Jul 8, 2019

thilp Jul 8, 2019 •

edited

Loading

hjacobs Jul 8, 2019

nolar Jul 9, 2019

nolar Jul 9, 2019 •

edited

Loading

hjacobs Jul 8, 2019

nolar Jul 9, 2019

hjacobs Jul 8, 2019

hjacobs Jul 9, 2019

codecov-io commented Jul 9, 2019 •

edited

Loading

nolar commented Jul 9, 2019

thilp left a comment

thilp Jul 9, 2019

nolar commented Jul 9, 2019

		from kopf.k8s import config


		# TODO: this mixin has to be migrated into pykube-ng itself.

		kind = "CustomResourceDefinition"


		def _make_cls(resource) -> Type[APIObject]:



		# TODO: this mixin has to be migrated into pykube-ng itself.
		class APIObject(pykube.objects.APIObject):

Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) #110

Switch to pykube-ng to handle arbitrary resources (pods, jobs, etc) #110

Conversation

nolar commented Jun 14, 2019 • edited Loading

Description

Types of Changes

Todos

Review

zincr bot commented Jun 14, 2019 • edited Loading

🤖 zincr found 0 problems , 2 warnings

Large Commits

Dependency Licensing

angelbarrera92 commented Jun 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolar Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolar commented Jul 8, 2019

hjacobs commented Jul 8, 2019

thilp Jul 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolar Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jul 9, 2019 • edited Loading

Codecov Report

nolar commented Jul 9, 2019

thilp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolar commented Jul 9, 2019

nolar commented Jun 14, 2019 •

edited

Loading

zincr bot commented Jun 14, 2019 •

edited

Loading

nolar Jul 9, 2019 •

edited

Loading

thilp Jul 8, 2019 •

edited

Loading

nolar Jul 9, 2019 •

edited

Loading

codecov-io commented Jul 9, 2019 •

edited

Loading