Updating to use IndexInformer #92

jrnt30 · 2017-08-12T18:16:27Z

NOTE: This is the start of a rebase/rework for #32. There ended up being some significant changes that made it difficult to integrate the changes easily and I ended up moving around a bit of the logic from store and server in an attempt to make it a bit more reusable and testable.
Very open to feedback but wanted to get some eyes on this from @jtblin and @mikekap if they have time to ensure I'm heading down the right path.

I have done some initial testing in our staging cluster but need to spend some more time next week going through some of the edge cases on in server/ and test some cross-account roles tested

Removed reliance on the store for maintain state and
migrated to the IndexInformer. This now will only cache
"active" pods (no failed/completed pods will be returned)
Extracted role based fallback logic to it's own processor
and removed from the Server/Event handlers
Migrated event handlers to the k8s package
Added in processor tests for the new RoleProcessor

- Removed reliance on the store for maintain state and migrated to the IndexInformer. This now will only cache "active" pods (no failed/completed pods will be returned) - Extracted role based fallback logic to it's own processor and removed from the Server/Event handlers - Migrated event handlers to the k8s package - Moved the `cmd` package back to the base to allow for easy builds and acknowlding the fact that the project itself is the unit of deployment and it's unlikely this would be used as a library in and of itself - Added in processor tests for the new RoleProcessor

coveralls · 2017-08-12T18:23:48Z

Coverage increased (+18.3%) to 28.205% when pulling 83e2771 on jrnt30:index-refactor into 12fa75c on jtblin:master.

jtblin · 2017-08-17T09:48:50Z

Thanks @jrnt30, can you please revert this:

Moved the cmd package back to the base to allow for easy builds
and acknowledging the fact that the project itself is the unit of
deployment and it's unlikely this would be used as a library in and of
itself

I will start reviewing when this is done. It's the "standard" directory structure for go projects and I would like to keep it as-is. Moreover it creates unnecessary noise in the PR making it harder to review.

Conflicts: namespace.go pod.go server/server.go store/store.go

jrnt30 · 2017-08-17T10:38:20Z

@jtblin Sure thing, that is done and conflicts merged with master.

jtblin

Thank you. Just a few nitpicks for now, I will be travelling so will continue the review during the weekend. Also it seems there are some tests failing in the build.

jtblin · 2017-08-17T10:48:37Z

k8s/k8s.go

@@ -3,6 +3,8 @@ package k8s
 import (
 	"time"

+	"fmt"


Please put before "time" dep.

jtblin · 2017-08-17T10:52:22Z

server/server.go

 	"github.com/jtblin/kube2iam/iam"
 	"github.com/jtblin/kube2iam/k8s"
-	"github.com/jtblin/kube2iam/store"
+	"github.com/jtblin/kube2iam/processor"
+	"k8s.io/client-go/tools/cache"


Please move the k8s.io dep to the other external deps i.e. after log "github.com/sirupsen/logrus".

jtblin · 2017-08-17T10:53:56Z

processor/processor.go

+package processor
+
+import (
+	"fmt"


Please keep deps orderered:

Std lib e.g. fmt, encoding/json

external deps

kube2iam deps

jrnt30 · 2017-08-18T00:49:53Z

Thanks for the feedback. The imports structure makes a lot of sense, those are cleaned up.

The tests themselves were an oversight when I moved main back into the cmd/ package but didn't change the Coveralls configuration back.

That's been fixed as well.

coveralls · 2017-08-19T17:27:36Z

Coverage increased (+18.3%) to 28.205% when pulling bedae78 on jrnt30:index-refactor into e9db60b on jtblin:master.

jtblin

Thanks for your patience. Few more changes requested, it's a huge PR so it's expected, almost there. Good work overall! 👍

jtblin · 2017-08-21T09:54:28Z

cover.sh

@@ -11,13 +11,12 @@ echo "mode: count" > coverage.out
 # Initialize error tracking
 ERROR=""

-declare -a packages=('' \
-    'cmd' \
+declare -a packages=('cmd' \


I think that precents root files to be tested. Please follow the original usage and add "processor" after "k8s".

Since there are no *.go files now, Coveralls will complain if the '' is left in. Moved the other to be sorted.

jtblin · 2017-08-21T09:54:56Z

Makefile

@@ -26,10 +26,10 @@ setup:
 	gometalinter --install --update
 	glide install --strip-vendor

-build: *.go fmt


Why changing that ooc? Was that creating any issue for you?

Since we removed the root *.go files it didn't trigger properly.

jtblin · 2017-08-21T09:55:21Z

k8s/k8s.go

@@ -13,12 +14,18 @@ import (

 const (
 	// Resync period for the kube controller loop.
-	resyncPeriod = 30 * time.Minute


Please order alphabetically.

jtblin · 2017-08-21T09:57:03Z

k8s/k8s.go

 )

 // Client represents a kubernetes client.
 type Client struct {
 	*kubernetes.Clientset
+	podIndexer          cache.Indexer


Please order alphabetically. It makes finding things easier.

namespaceController *cache.Controller namespaceIndexer cache.Indexer podController *cache.Controller podIndexer cache.Indexer

jtblin · 2017-08-21T10:01:26Z

k8s/k8s.go

@@ -75,5 +137,7 @@ func NewClient(host, token string, insecure bool) (*Client, error) {
 	if err != nil {
 		return nil, err
 	}
-	return &Client{client}, nil
+	k8s := &Client{}


This could be simplify to: return &Client{Clientset: client}, nil

jtblin · 2017-08-23T06:47:12Z

processor/processor.go

+	ar := GetNamespaceRoleAnnotation(ns, r.namespaceKey)
+	for _, role := range ar {
+		if r.iam.RoleARN(role) == roleArn {
+			log.Debugf("Role:%s on namespace:%s found.", roleArn, namespace)


Nit: space after :

jtblin · 2017-08-23T06:49:14Z

processor/processor.go

+			return true
+		}
+	}
+	log.Warnf("Role: [%s] on namespace: [%s] not found.", roleArn, namespace)


I think the [] are unnecessary and better use fields for structured logging. See *Fields convenience methods in other places.

jtblin · 2017-08-23T06:53:27Z

processor/processor_test.go

@@ -0,0 +1,241 @@
+package processor


Thanks for adding all these tests! 👍

There's no test for GetRoleMapping, maybe not strictly necessary though as you're testing the other methods but if not too painful could be nice.

jtblin · 2017-08-23T06:54:50Z

server/server.go

+	}
+
+	if !synced {
+		log.Fatalf("Attempted to wait for caches to be synced for [%d] however it is not done.  Giving up.", defaultCacheSyncAttempts)


Nit ... for %d attempts, however ...

jtblin · 2017-08-23T06:55:51Z

server/server.go

+
+	synced := false
+	for i := 0; i < defaultCacheSyncAttempts && !synced; i++ {
+		synced = cache.WaitForCacheSync(nil, podSynched, namespaceSynched)


Nice, that should prevent the race condition for default role? 👍

It was more around ensuring that both the pods and namespaces were indexed to ensure that you couldn't "sneak in" a role. This only happens @ startup time, so there is the possibility that you could get the default role immediately if the new pod hasn't been indexed.

What this PR aims to solve is a permanent caching of those incorrect roles.

dadux · 2017-08-23T07:31:04Z

We've tried running of this branch in our dev environments, but still get a lot of "default role" under load (our build-engineering team spinning up 100s of concurrent jobs)

jtblin · 2017-08-23T08:33:03Z

Thanks for the info @dadux, that's annoying. Looking at the code, it only falls back to the default role if the pod is found by its ip, there is no annotation, and there is a default role set. So if it's getting the default role when it shouldn't, it can only be because the cache has not been updated yet.

The solutions I can think of are:

Make a synchronous API call in case of fallback to ensure an annotation hasn't been set. There would be a latency impact obviously and a new code path to manage.
Move the fallback to the default role down the chain so that we return an error from extractRoleARN. That will trigger the exponential backoff operation retry and should be able to catch the updated annotation. Latency impact as well but at least no need for additional synchronous API call.
Remove the default role option as it is not reliable.

Anyhow let's not try to tackle that in this PR for now. We can look into it after.

jrnt30 · 2017-08-23T14:10:56Z

@jtblin Thanks for the guidance and constructive feedback, it's very helpful. I have addressed a majority of these issues. The processor renaming I have left as is for now and will come up with something better.

coveralls · 2017-08-24T07:21:18Z

Coverage increased (+8.3%) to 18.212% when pulling 12a6bcc on jrnt30:index-refactor into e9db60b on jtblin:master.

jtblin

Sorry I got distracted. I think it's the last nit apart from package naming. I'd suggest just going with "mappings" for now or something like that.
I was also waiting for @dadux to confirm that everything was looking good with the binary built from this branch.

jtblin · 2017-08-24T07:14:53Z

Makefile

@@ -29,7 +29,7 @@ setup:
 build: *.go fmt
 	go build -o build/bin/$(ARCH)/$(BINARY_NAME) $(GOBUILD_VERSION_ARGS) github.com/jtblin/$(BINARY_NAME)/cmd

-build-race: *.go fmt


I think we can change this one back as well now.

jtblin · 2017-08-24T07:19:08Z

processor/processor.go

+	namespaceKey         string
+	namespaceRestriction bool
+	iam                  *iam.Client
+	kubeStore            store


Sorry I meant changing the variable name from kubeStore to store in the struct as well pls.

jrnt30 · 2017-08-31T11:20:35Z

I also know that @placydo was looking at it a bit. Are your experiencing ok so far?

coveralls · 2017-08-31T11:22:04Z

Changes Unknown when pulling c8e937a on jrnt30:index-refactor into ** on jtblin:master**.

coveralls · 2017-08-31T11:33:41Z

Changes Unknown when pulling 775c7cf on jrnt30:index-refactor into ** on jtblin:master**.

jtblin · 2017-09-01T08:53:43Z

I've pushed docker image jtblin/kube2iam:dev for convenience if someone wants to try in its cluster. Would really appreciate some feedback given the size of the PR.

placydo · 2017-09-01T09:47:48Z

Hey. Currently I am in middle of something. I will try to pull this image today and give it a try + test it over the weekend. Will come back with feedback asap. Thanks guyz!

jtblin · 2017-09-04T07:23:51Z

@struz did you guys see any issue running this on our cluster?

placydo · 2017-09-11T12:56:23Z

@jtblin @jrnt30 We have not seen this issue anymore. Everything works as expected!

Fsero · 2017-09-11T16:57:00Z

mappings/mapper.go

+	return false
+}
+
+// DumpDebugInfo outputs all the roles by IP addresr.


nit: s/addresr/address/

jtblin · 2017-09-18T00:58:40Z

Thanks a ton again @jrnt30 for the great work. This is now merged and released as 0.8.0.

Justin Nauman added 2 commits August 13, 2017 18:39

Ensuring caches have been fully synced prior to starting server

a3bff93

Addressing the gometalinter checks

0bd0ea7

Justin Nauman added 2 commits August 17, 2017 05:24

Reverting cmd/ package restructure

58823c7

Merge remote-tracking branch 'origin/master' into index-refactor

9de4bac

Conflicts: namespace.go pod.go server/server.go store/store.go

jtblin requested changes Aug 18, 2017

View reviewed changes

Import cleanup

bedae78

jrnt30 mentioned this pull request Aug 19, 2017

ExternalDNS + kube2iam integration problem with default role kubernetes-sigs/external-dns#292

Closed

jtblin requested changes Aug 23, 2017

View reviewed changes

Address PR comments

12a6bcc

jtblin requested changes Aug 31, 2017

View reviewed changes

Processor package renaming

c8e937a

jrnt30 force-pushed the index-refactor branch from 02388d2 to c8e937a Compare August 31, 2017 11:14

Missed a comment

775c7cf

jtblin mentioned this pull request Sep 1, 2017

Bind IAM credentials to ServiceAccounts, not directly to Pods #79

Open

jtblin changed the title ~~WIP: Updating to use IndexInformer~~ Updating to use IndexInformer Sep 4, 2017

This was referenced Sep 4, 2017

Kill the store #32

Closed

Incorrect ip to namespace table for pods in a failed state. #95

Closed

placydo mentioned this pull request Sep 11, 2017

default-role race condition #80

Open

Fsero reviewed Sep 11, 2017

View reviewed changes

mappings/mapper.go

return false

}

// DumpDebugInfo outputs all the roles by IP addresr.

Copy link

Fsero Sep 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: s/addresr/address/

jtblin approved these changes Sep 17, 2017

View reviewed changes

jtblin merged commit c5ef1ec into jtblin:master Sep 17, 2017

jrnt30 mentioned this pull request Sep 22, 2017

DefaultRole being returned for new pods under load #100

Open

jrnt30 mentioned this pull request Sep 30, 2017

kube2iam loses pod's credentials? #102

Closed

Updating to use IndexInformer #92

Updating to use IndexInformer #92

Conversation

jrnt30 commented Aug 12, 2017 • edited Loading

coveralls commented Aug 12, 2017

jtblin commented Aug 17, 2017

jrnt30 commented Aug 17, 2017

jtblin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrnt30 commented Aug 18, 2017

coveralls commented Aug 19, 2017

jtblin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dadux commented Aug 23, 2017

jtblin commented Aug 23, 2017

jrnt30 commented Aug 23, 2017

coveralls commented Aug 24, 2017

jtblin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrnt30 commented Aug 31, 2017

coveralls commented Aug 31, 2017

coveralls commented Aug 31, 2017

jtblin commented Sep 1, 2017

placydo commented Sep 1, 2017

jtblin commented Sep 4, 2017

placydo commented Sep 11, 2017

Choose a reason for hiding this comment

jtblin commented Sep 18, 2017

jrnt30 commented Aug 12, 2017 •

edited

Loading