Add support for cross account roles #4

atward · 2019-05-13T03:42:56Z

This adds the ability to query buckets outside the current account. This is needed as aws-go-sdk has no credentials config support for assuming roles when using instance profiles.

ribbybibby · 2019-05-13T11:44:35Z

Thanks for the PR @atward!

This is needed as aws-go-sdk has no credentials config support for assuming roles when using instance profiles.

Have you tried configuring a profile as detailed on this page https://docs.aws.amazon.com/cli/latest/topic/config-vars.html?

# In ~/.aws/config
[profile crossaccount]
role_arn=arn:aws:iam:...
credential_source=Ec2InstanceMetadata

I'm mostly just interested as I have ran into the same problem with instance profiles and assumed roles myself in the past and I'd never seen this option before. It looks to me like aws-sdk-go should support it as of this PR: aws/aws-sdk-go#2201.

I don't have quick access to an AWS environment where I can test this, so I can't check it myself right now.

Even if it does work, I do understand that setting up an ~/.aws/config file is probably less desirable than a command line flag in a lot of cases. However, as a rule, I try to avoid adding AWS credential options that can be set outside of the program, so I'm a little hesitant here.

atward · 2019-05-14T01:11:01Z

I was not aware of pr/2201 I was going off issues/1019 which was still open.

Config works as per the linked python documentation given AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount

atward · 2019-05-15T04:26:56Z

I spoke too soon.

When I compile s3_exporter it works as described. Using the pre-compiled 0.1.0 binary results in AccessDenied.

I'm unsure what's causing the problem but could we get an updated release?

atward · 2019-05-15T04:32:46Z

Logs

go get github.com/ribbybibby/s3_exporter
cd ~/go/src/github.com/ribbybibby/s3_exporter/
make

>> checking code style
mkdir -p /root/go/bin
curl -s -L https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 > /root/go/bin/staticcheck
>> running staticcheck
chmod +x /root/go/bin/staticcheck
GO111MODULE=on go list -e -compiled -test=true -export=false -deps=true -find=false -tags= -- ./... > /dev/null
go: finding github.com/prometheus/common v0.3.0
go: finding github.com/prometheus/client_golang v0.9.2
go: finding github.com/aws/aws-sdk-go v1.19.11
go: finding golang.org/x/text v0.3.0
go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: finding github.com/prometheus/client_golang v0.9.1
go: finding github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding github.com/gogo/protobuf v1.1.1
go: finding golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
go: finding github.com/golang/protobuf v1.2.0
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/julienschmidt/httprouter v1.2.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
go: finding github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
go: finding golang.org/x/net v0.0.0-20181201002055-351d144fa1fc
go: finding github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding golang.org/x/net v0.0.0-20181114220301-adae6a3d119a
go: finding github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: finding github.com/sirupsen/logrus v1.2.0
go: finding github.com/go-logfmt/logfmt v0.3.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/go-stack/stack v1.8.0
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/stretchr/objx v0.1.1
go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/aws/aws-sdk-go v1.19.11
go: downloading github.com/prometheus/common v0.3.0
go: downloading github.com/prometheus/client_golang v0.9.2
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: downloading github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: downloading github.com/sirupsen/logrus v1.2.0
go: downloading golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/golang/protobuf v1.2.0
go: downloading github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: downloading github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: downloading github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5
go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
GO111MODULE=on /root/go/bin/staticcheck -ignore "" ./...
>> running check for unused/missing packages in go.mod
GO111MODULE=on go mod tidy
go: downloading github.com/stretchr/testify v1.2.2
go: downloading golang.org/x/net v0.0.0-20181201002055-351d144fa1fc
go: downloading golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading golang.org/x/text v0.3.0
>> running check for unused packages in vendor/
GO111MODULE=on go mod vendor
curl -s -L https://github.com/prometheus/promu/releases/download/v0.2.0/promu-0.2.0.linux-amd64.tar.gz | tar -xvzf - -C /tmp/tmp.hAlGG1rwP9
promu-0.2.0.linux-amd64/
promu-0.2.0.linux-amd64/LICENSE
promu-0.2.0.linux-amd64/NOTICE
promu-0.2.0.linux-amd64/promu
mkdir -p /root/go/bin
cp /tmp/tmp.hAlGG1rwP9/promu-0.2.0.linux-amd64/promu /root/go/bin/promu
rm -r /tmp/tmp.hAlGG1rwP9
>> building binaries
GO111MODULE=on /root/go/bin/promu build --prefix /root/go/src/github.com/ribbybibby/s3_exporter
 >   s3_exporter

$ AWS_REGION=ap-southeast-2 AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount ./s3_exporter --log.level=debug 
INFO[0000] Starting s3_exporter (version=, branch=, revision=)  source="s3_exporter.go:164"
INFO[0000] Build context (go=go1.11.5, user=, date=)     source="s3_exporter.go:165"
INFO[0000] Listening on :9340                            source="s3_exporter.go:182"

When using precompiled binaries v0.1.0

$ AWS_REGION=ap-southeast-2 AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount /usr/local/sbin/s3_exporter --log.level=debug
INFO[0000] Starting s3_exporter (version=, branch=, revision=)  source="s3_exporter.go:164"
INFO[0000] Build context (go=go1.8.3, user=, date=)      source="s3_exporter.go:165"
INFO[0000] Listening on :9340                            source="s3_exporter.go:182"
ERRO[0012] AccessDenied: Access Denied
	status code: 403, request id: 32DBLAHA0F, host id: WA2nTjs/STUFFKD5Ph8qk=  source="s3_exporter.go:82"

make tarball attached for comparison: s3_exporter-0.1.0.linux-amd64.tar.gz

ribbybibby · 2019-05-15T17:36:21Z

Hi @atward. I've released 0.2.0 - can you please check if it works for you.

atward · 2019-05-15T23:55:53Z

Closed by v0.2.0 release

  Service s3_exporter-crossaccount
     ✔  should be running
  File /etc/systemd/system/s3_exporter-crossaccount.service
     ✔  should exist
     ✔  content should include "AWS_REGION=ap-southeast-2"
     ✔  content should include "AWS_SDK_LOAD_CONFIG=true"
     ✔  content should include "AWS_PROFILE=crossaccount"
  Processes s3_exporter
     ✔  entries.length should eq 1
     ✔  commands.first should include "--web.listen-address localhost:9342"
  http GET on localhost:9342/metrics
     ✔  status should eq 200
     ✔  body should include "go_memstats"
  http GET on localhost:9342/probe
     ✔  status should eq 200
     ✔  body should include "s3_list_success{bucket=\"\",prefix=\"\"} 0"
  http GET on localhost:9342/probe?bucket=mybucket
     ✔  status should eq 200
     ✔  body should include "s3_list_success{bucket=\"mybucket\",prefix=\"\"} 1"

Add support for cross account roles

f709b2b

atward closed this May 14, 2019

atward reopened this May 15, 2019

atward closed this May 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for cross account roles #4

Add support for cross account roles #4

atward commented May 13, 2019

ribbybibby commented May 13, 2019

atward commented May 14, 2019

atward commented May 15, 2019

atward commented May 15, 2019 •

edited

Loading

ribbybibby commented May 15, 2019

atward commented May 15, 2019

Add support for cross account roles #4

Add support for cross account roles #4

Conversation

atward commented May 13, 2019

ribbybibby commented May 13, 2019

atward commented May 14, 2019

atward commented May 15, 2019

atward commented May 15, 2019 • edited Loading

ribbybibby commented May 15, 2019

atward commented May 15, 2019

atward commented May 15, 2019 •

edited

Loading