Add support for IAM role credentials #104

Igorshp · 2015-03-10T11:07:01Z

This is a patch for PR by @daanemanz - #68
The problem was that fetch_credentials does not exist in the context and would always fall back to rescue block.

The original issue is #55

Igorshp · 2015-03-10T14:05:19Z

noooo, line's too long... great

Igorshp · 2015-03-10T14:49:03Z

Yes :) passes the tests finaly

Igorshp · 2015-03-12T12:27:11Z

@fnichol do you have any comments?

Igorshp · 2015-03-12T12:34:50Z

After a bit of throughout, I've reordered the precedence of aim_role_credentials.
Using aim_role should be a fallback, not a priority.
This will allow a node with aim role to still use custom keys if needed.

willejs · 2015-03-12T17:52:18Z

👍

iiro · 2015-03-16T19:07:46Z

👍

Igorshp · 2015-03-18T14:12:14Z

Now using this on our jenkins server.
Works a treat

tyler-ball · 2015-03-25T20:40:39Z

Why do you prefer this PR over #68 ? I made a comment in that PR that would apply here too. You also should not be using an include at the top level - this includes that module at the Kernel level (which is bad).

Igorshp · 2015-03-25T20:59:30Z

Thanks for reply @tyler-ball
unfortuantely #68 has a little bug, namely the fact that
fetch_credentials() is not availalbe in the context and needs to be called with full namespace like so:
Fog::AWS::CredentialFetcher::ServiceMethods.fetch_credentials()

This PR builds on top of #68. Unfortuantely @daanemanz deleted his fork so I couldn't contribute to original PR.

Where should include go? just above the iam_cred request?

Igorshp · 2015-03-25T21:01:07Z

I'll integrate your comments from the other PR.

tyler-ball · 2015-03-26T21:48:52Z

@Igorshp When you include the module you add fetch_credentials as an instance-level method. If you extended the module then fetch_credentials would be a class-level method and callable directly in the class - like #68 was doing. But if you use my suggestion from the other PR and call fetch_credentials from within a method, it will be invoked on the instance. This won't require the full namespace.

The include can go on the line directly below the class declaration.

Igorshp · 2015-03-30T13:56:50Z

@tyler-ball i've implemented your comments into the PR.
It definitely looks a lot cleaner now, thanks!

The iam_creds method had to be come static though or it woudn't be allowed to be called during class initialisation.

Any preference of extend/include? Extend works well right now and as you said doesn't require full namespace prefix, include still does (even when calling from method).

As mentioned in test-kitchen#55. This change will add support for using IAM temporary credentials when creating EC2 instances. Currently the credentials will have to be set in the environment, while an EC2 instance that's set up with an IAM profile can fetch its temporary credentials from the metadata server.

tweak for PR by daanemanz test-kitchen#68 he has since deleted his fork, so I can't submit a PR to him.

using aim_role should be a fallback, not a priority. This will allow a node with aim role to still use custom keys if needed

hooray for passing checks!

Igorshp · 2015-03-30T14:16:39Z

rebased on master....

tyler-ball · 2015-03-30T16:52:01Z

lib/kitchen/driver/ec2.rb

+        @iam_creds ||= begin
+         fetch_credentials(use_iam_profile:true)
+       rescue
+         debug("fetch_credentials failed with exception #{e.message}:#{e.backtrace.join("\n")}")


The var e doesn't exist here because you haven't declared it. The rescue line should look like rescue RuntimeError => e

tyler-ball · 2015-03-30T16:53:04Z

I had 1 more comment, and it looks like Travis is failing with Tailor code style errors. Fix those 2 things and this should be good to merge!

JamesAwesome · 2015-03-31T04:21:46Z

👍

Igorshp · 2015-03-31T16:24:09Z

@tyler-ball bah, totally skipped over that. Thanks for picking it up!
Will get the tests sorted shortly

Igorshp · 2015-03-31T16:38:13Z

👍

Add support for IAM role credentials

tyler-ball · 2015-04-06T20:57:14Z

Thanks for this PR!

Igorshp · 2015-04-06T20:57:31Z

Thanks for the merge @tyler-ball!

otanner · 2015-04-08T15:04:57Z

When using this on a local machine I am experiencing an issue when the 'kitchen converge' is always crashing because the fetch_credentials() can't get the AWS metadata from 169.254.169.254. Because of this it makes kitchen-ec2 unusable if it is not ran on an EC2 instance.

[fog][WARNING] Unable to fetch credentials: connect timeout reached
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #create action: [super: no superclass method `fetch_credentials' for Kitchen::Driver::Ec2:Class]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Is this happening because self.iam_creds is always calling fetch_credentials() even if is not needed?

JamesAwesome · 2015-04-08T15:07:40Z

~~Hmmm, something that my original method for dealing with iam roles (via a config attribute) in #107 would have corrected but autodetection messes up.~~

Scratch that, i catch NoMethod in #107 so it wouldn't be an issue.

Igorshp · 2015-04-08T15:57:01Z

@otanner how are you running kitchen-ec2 plugin localy?

iiro · 2015-04-08T16:06:57Z

Hi guys,

I'm having the same problem here.

I'm using ChefDK's (0.4.0) Test-Kitchen just by running "kitchen converge" - and I can repeat this behaviour by upgrading to latest 0.8.1-dev as well.

otanner · 2015-04-08T16:13:05Z

@Igorshp I am setting the driver name to ec2 and putting the needed AWS configuration in place (api keys, region, etc). With the kitchen-ec2 0.8.0 the same configuration works. I'm running ChefDK's Test-Kitchen on OS X.

Igorshp · 2015-04-08T17:14:46Z

@otanner @iiro could you provide some more info, i'm struggling to reproduce the problem.

Using kitchen-ec2 0.8.1.dev from git@github.com:test-kitchen/kitchen-ec2.git (at master)

Using latest version from master, osx, tried with and without chefdk.

Kitchen successfully creates an ec2 instance.

tyler-ball · 2015-04-08T20:10:54Z

@otanner can you check your kitchen logs and see what is the error raised by Fog? The rescue should be catching that if 169.254.169.254 is unreachable and memoizing iam_creds as {}

otanner · 2015-04-08T22:17:24Z

@tyler-ball @Igorshp Here is an unsuccessful run from kitchen.log. Notice the one minute delay between starting the kitchen and getting the error as it waits for the network timeout. So even if the rescue would (which it apparently does not) work and catch the error in the end it would cause unnecessary one minute delay on each run.

Can there be a bug in Fog as well because the code falls back with 'super' but the parent class doesn't have a suitable method to run?

I, [2015-04-08T17:51:47.201303 #69286]  INFO -- Kitchen: -----> Starting Kitchen (v1.3.1)
I, [2015-04-08T17:51:50.526005 #69286]  INFO -- Kitchen: -----> Creating <testbox-ubuntu-1404>...
E, [2015-04-08T17:52:50.547601 #69286] ERROR -- Kitchen: ------Exception-------
E, [2015-04-08T17:52:50.548088 #69286] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2015-04-08T17:52:50.548139 #69286] ERROR -- Kitchen: Message: Failed to complete #create action: [super: no superclass method `fetch_credentials' for Kitchen::Driver::Ec2:Class]
E, [2015-04-08T17:52:50.548202 #69286] ERROR -- Kitchen: ---Nested Exception---
E, [2015-04-08T17:52:50.548223 #69286] ERROR -- Kitchen: Class: NoMethodError
E, [2015-04-08T17:52:50.548242 #69286] ERROR -- Kitchen: Message: super: no superclass method `fetch_credentials' for Kitchen::Driver::Ec2:Class
E, [2015-04-08T17:52:50.548261 #69286] ERROR -- Kitchen: ------Backtrace-------
E, [2015-04-08T17:52:50.548280 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/fog-aws-0.1.2/lib/fog/aws/credential_fetcher.rb:24:in `rescue in fetch_credentials'
E, [2015-04-08T17:52:50.548299 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/fog-aws-0.1.2/lib/fog/aws/credential_fetcher.rb:9:in `fetch_credentials'
E, [2015-04-08T17:52:50.548318 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:103:in `iam_creds'
E, [2015-04-08T17:52:50.548337 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:51:in `block in <class:Ec2>'
E, [2015-04-08T17:52:50.548355 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/lazy_hash.rb:116:in `call'
E, [2015-04-08T17:52:50.548373 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/lazy_hash.rb:116:in `proc_or_val'
E, [2015-04-08T17:52:50.548391 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/lazy_hash.rb:76:in `[]'
E, [2015-04-08T17:52:50.548410 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:177:in `connection'
E, [2015-04-08T17:52:50.548428 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:186:in `create_server'
E, [2015-04-08T17:52:50.548446 #69286] ERROR -- Kitchen: /Users/otanner/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:124:in `create'
E, [2015-04-08T17:52:50.548464 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:352:in `public_send'
E, [2015-04-08T17:52:50.548482 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:352:in `block in perform_action'
E, [2015-04-08T17:52:50.548500 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:416:in `call'
E, [2015-04-08T17:52:50.548534 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:416:in `synchronize_or_call'
E, [2015-04-08T17:52:50.548556 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:381:in `block in action'
E, [2015-04-08T17:52:50.548575 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
E, [2015-04-08T17:52:50.548593 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:380:in `action'
E, [2015-04-08T17:52:50.548611 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:352:in `perform_action'
E, [2015-04-08T17:52:50.548629 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:303:in `create_action'
E, [2015-04-08T17:52:50.548647 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:292:in `block in transition_to'
E, [2015-04-08T17:52:50.548703 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:291:in `each'
E, [2015-04-08T17:52:50.548722 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:291:in `transition_to'
E, [2015-04-08T17:52:50.548740 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:130:in `converge'
E, [2015-04-08T17:52:50.548758 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/command.rb:176:in `public_send'
E, [2015-04-08T17:52:50.548776 #69286] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/command.rb:176:in `block (2 levels) in run_action'
E, [2015-04-08T17:52:50.548793 #69286] ERROR -- Kitchen: ----------------------

tyler-ball · 2015-04-08T23:24:30Z

@otanner I reproduced your error locally - we're working on it in #107

tyler-ball added the in progress label Mar 25, 2015

drosenstark and others added 9 commits March 30, 2015 15:15

Make all the tests pass

b65a992

Use AIM role for authentication

0142ce3

tweak for PR by daanemanz test-kitchen#68 he has since deleted his fork, so I can't submit a PR to him.

formated to pass travis

6a634e1

reindented

a32c18f

reordered aim creds priority

2772408

using aim_role should be a fallback, not a priority. This will allow a node with aim role to still use custom keys if needed

removed trailing whitespaces

e3ffa8b

hooray for passing checks!

moved include to the top of the file

b4c3e10

updated based on comments from @tyler-ball

54c462f

Igorshp force-pushed the aim_roles branch from 4b0a001 to 54c462f Compare March 30, 2015 14:16

tyler-ball reviewed Mar 30, 2015
View reviewed changes

fixing travis errors

c7485de

tyler-ball added a commit that referenced this pull request Apr 6, 2015

Merge pull request #104 from Igorshp/aim_roles

8325c9d

Add support for IAM role credentials

tyler-ball merged commit 8325c9d into test-kitchen:master Apr 6, 2015

tyler-ball removed the in progress label Apr 6, 2015

JamesAwesome mentioned this pull request Apr 7, 2015

Add explicit option for using iam profile for authentication #107

Merged

tyler-ball mentioned this pull request Apr 10, 2015

Add support for IAM credentials #68

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for IAM role credentials #104

Add support for IAM role credentials #104

Igorshp commented Mar 10, 2015

Igorshp commented Mar 10, 2015

Igorshp commented Mar 10, 2015

Igorshp commented Mar 12, 2015

Igorshp commented Mar 12, 2015

willejs commented Mar 12, 2015

iiro commented Mar 16, 2015

Igorshp commented Mar 18, 2015

tyler-ball commented Mar 25, 2015

Igorshp commented Mar 25, 2015

Igorshp commented Mar 25, 2015

tyler-ball commented Mar 26, 2015

Igorshp commented Mar 30, 2015

Igorshp commented Mar 30, 2015

tyler-ball Mar 30, 2015

tyler-ball commented Mar 30, 2015

JamesAwesome commented Mar 31, 2015

Igorshp commented Mar 31, 2015

Igorshp commented Mar 31, 2015

tyler-ball commented Apr 6, 2015

Igorshp commented Apr 6, 2015

otanner commented Apr 8, 2015

JamesAwesome commented Apr 8, 2015

Igorshp commented Apr 8, 2015

iiro commented Apr 8, 2015

otanner commented Apr 8, 2015

Igorshp commented Apr 8, 2015

tyler-ball commented Apr 8, 2015

otanner commented Apr 8, 2015

tyler-ball commented Apr 8, 2015

Add support for IAM role credentials #104

Add support for IAM role credentials #104

Conversation

Igorshp commented Mar 10, 2015

Igorshp commented Mar 10, 2015

Igorshp commented Mar 10, 2015

Igorshp commented Mar 12, 2015

Igorshp commented Mar 12, 2015

willejs commented Mar 12, 2015

iiro commented Mar 16, 2015

Igorshp commented Mar 18, 2015

tyler-ball commented Mar 25, 2015

Igorshp commented Mar 25, 2015

Igorshp commented Mar 25, 2015

tyler-ball commented Mar 26, 2015

Igorshp commented Mar 30, 2015

Igorshp commented Mar 30, 2015

tyler-ball Mar 30, 2015

Choose a reason for hiding this comment

tyler-ball commented Mar 30, 2015

JamesAwesome commented Mar 31, 2015

Igorshp commented Mar 31, 2015

Igorshp commented Mar 31, 2015

tyler-ball commented Apr 6, 2015

Igorshp commented Apr 6, 2015

otanner commented Apr 8, 2015

JamesAwesome commented Apr 8, 2015

Igorshp commented Apr 8, 2015

iiro commented Apr 8, 2015

otanner commented Apr 8, 2015

Igorshp commented Apr 8, 2015

tyler-ball commented Apr 8, 2015

otanner commented Apr 8, 2015

tyler-ball commented Apr 8, 2015