Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Socket permission on S3 Repository creation #23271

Closed
dadoonet opened this issue Feb 20, 2017 · 2 comments
Closed

Socket permission on S3 Repository creation #23271

dadoonet opened this issue Feb 20, 2017 · 2 comments
Assignees
@Tim-Brooks
Labels
>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >regression

Comments

@dadoonet
Copy link
Member

Elasticsearch version: 6.0.0-alpha1-SNAPSHOT (872412f)
Plugins installed: [ repository-s3 ]
JVM version: 1.8.0_121
OS version: MacOS
Description of the problem including expected versus actual behavior:

Registering an S3 repository is now failing on master branch.

Steps to reproduce:

  1. Install the distribution, and the repository-s3 plugin
  2. Create the keystore and s3 credentials:
$ bin/elasticsearch-keystore list
s3.client.my_config.access_key
s3.client.my_config.secret_key
  1. Starts elasticsearch
  2. Register a repository:
curl -X POST 'http://127.0.0.1:9200/_snapshot/backups?pretty=true&error_trace=true' -H 'Content-Type: application/json' -d '{
   "type":"s3",
   "settings":{
      "bucket": "BUCKET",
      "region": "eu-west-1"
   }
}'

Provide logs (if relevant):

[2017-02-20T19:54:01,577][INFO ][o.e.n.Node               ] [] initializing ...
[2017-02-20T19:54:01,640][INFO ][o.e.e.NodeEnvironment    ] [Qo1UYJ1] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [11.7gb], net total_space [464.7gb], spins? [unknown], types [hfs]
[2017-02-20T19:54:01,641][INFO ][o.e.e.NodeEnvironment    ] [Qo1UYJ1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-02-20T19:54:01,642][INFO ][o.e.n.Node               ] node name [Qo1UYJ1] derived from node ID [Qo1UYJ1SSKy_sEz70V1S3w]; set [node.name] to override
[2017-02-20T19:54:01,645][INFO ][o.e.n.Node               ] version[6.0.0-alpha1-SNAPSHOT], pid[69444], build[872412f/2017-02-20T18:38:31.989Z], OS[Mac OS X/10.12.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-02-20T19:54:01,646][WARN ][o.e.n.Node               ] version [6.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2017-02-20T19:54:03,618][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [aggs-matrix-stats]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [ingest-common]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-expression]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-mustache]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-painless]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [percolator]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [reindex]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [repository-url]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [transport-netty4]
[2017-02-20T19:54:03,620][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded plugin [repository-s3]
[2017-02-20T19:54:06,880][INFO ][o.e.n.Node               ] initialized
[2017-02-20T19:54:06,880][INFO ][o.e.n.Node               ] [Qo1UYJ1] starting ...
[2017-02-20T19:54:07,495][INFO ][o.e.t.TransportService   ] [Qo1UYJ1] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017-02-20T19:54:10,593][INFO ][o.e.c.s.ClusterService   ] [Qo1UYJ1] new_master {Qo1UYJ1}{Qo1UYJ1SSKy_sEz70V1S3w}{ByBFhbOGTr2rsilGvWAygQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-02-20T19:54:10,637][INFO ][o.e.h.n.Netty4HttpServerTransport] [Qo1UYJ1] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-02-20T19:54:10,652][INFO ][o.e.n.Node               ] [Qo1UYJ1] started
[2017-02-20T19:54:10,654][INFO ][o.e.g.GatewayService     ] [Qo1UYJ1] recovered [0] indices into cluster_state
[2017-02-20T19:54:24,971][WARN ][o.e.r.RepositoriesService] [Qo1UYJ1] failed to create repository [s3][backups]
java.security.AccessControlException: access denied ("java.net.SocketPermission" "169.254.169.254:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_121]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:510) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:91) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) ~[?:?]
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:70) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:345) ~[?:?]
        at org.elasticsearch.plugin.repository.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:71) ~[?:?]
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:679) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:658) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:617) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:1117) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-02-20T19:54:24,981][WARN ][o.e.r.RepositoriesService] [Qo1UYJ1] failed to create repository [backups]
org.elasticsearch.repositories.RepositoryException: [backups] failed to create repository
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:388) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:679) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:658) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:617) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:1117) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.security.AccessControlException: access denied ("java.net.SocketPermission" "169.254.169.254:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_121]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:510) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:91) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) ~[?:?]
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:70) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:345) ~[?:?]
        at org.elasticsearch.plugin.repository.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:71) ~[?:?]
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        ... 14 more
@dadoonet
Copy link
Member Author

Actually I did something wrong but still I'm getting this error.

Here is my mistake. I registered a client called my_config but I'm not using it when creating the repository.

With:

curl -X POST 'http://127.0.0.1:9200/_snapshot/backups?pretty=true&error_trace=true' -H 'Content-Type: application/json' -d '
{
   "type":"s3",
   "settings":{
      "client": "my_config",
      "bucket": "BUCKET",
      "region": "eu-west-1"
   }
}'

Everything works fine:

{
  "acknowledged" : true
}

So this appears only when no credentials are found. So when we switch to IAM Profiles instead.
In such case, AWS is trying to get the metadata from the instance by calling this URL 169.254.169.254:80.

I did not test but I believe that IAM Profile won't work.

Tim-Brooks added a commit to Tim-Brooks/elasticsearch that referenced this issue Feb 21, 2017
@Tim-Brooks
Wrap getCredentials() in a doPrivileged() block
f517e7a 
This commit fixes an issue that was missed in elastic#22534.
`AWSCredentialsProvider.getCredentials()` appears to potentially open a
socket connect. This operation needed to be wrapped in `doPrivileged()`.

This should fix issue elastic#23271.
@Tim-Brooks Tim-Brooks mentioned this issue Feb 21, 2017
Merged
Tim-Brooks added a commit that referenced this issue Feb 23, 2017
@Tim-Brooks
Wrap getCredentials() in a doPrivileged() block (#23297)
a4afc22 
This commit fixes an issue that was missed in #22534.
`AWSCredentialsProvider.getCredentials()` appears to potentially open a
socket connect. This operation needed to be wrapped in `doPrivileged()`.

This should fix issue #23271.
@dadoonet
Copy link
Member Author

Should be fixed by #23297

@clintongormley clintongormley added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Repository S3 labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Tim-Brooks Tim-Brooks

Labels
>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >regression
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clintongormley @dadoonet @Tim-Brooks