Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to start AWS discovery with 2.0.0-SNAPSHOT #12643

Closed
dadoonet opened this issue Aug 4, 2015 · 5 comments · Fixed by #12744
Closed

Unable to start AWS discovery with 2.0.0-SNAPSHOT #12643

dadoonet opened this issue Aug 4, 2015 · 5 comments · Fixed by #12744
Labels
blocker >bug :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs

Comments

@dadoonet
Copy link
Member

dadoonet commented Aug 4, 2015

I started 2 AWS instances using latest 2.0.0-SNAPSHOT I built locally.

When starting with defaults, everything is fine.

When changing elasticsearch.yml to:

discovery.type: ec2
discovery.ec2.tag.Name: dadoonet-test-2.0.0-SNAP

I get this error while launching:

[2015-08-04 16:37:18,363][ERROR][org.elasticsearch.bootstrap] Exception
NoClassSettingsException[Failed to load class setting [discovery.type] with value [ec2]]; nested: ClassNotFoundException[org.elasticsearch.discovery.ec2.Ec2DiscoveryModule];
    at org.elasticsearch.common.settings.Settings.loadClass(Settings.java:604)
    at org.elasticsearch.common.settings.Settings.getAsClass(Settings.java:592)
    at org.elasticsearch.discovery.DiscoveryModule.spawnModules(DiscoveryModule.java:53)
    at org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44)
    at org.elasticsearch.node.Node.<init>(Node.java:177)
    at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:157)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:272)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.discovery.ec2.Ec2DiscoveryModule
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.elasticsearch.common.settings.Settings.loadClass(Settings.java:602)
    ... 8 more

The cloud plugin contains needed libs I think:

[ec2-user@ip-10-0-0-113 cloud-aws]$ ll
total 5776
-rw-rw-r-- 1 ec2-user ec2-user  513834  4 août  16:32 aws-java-sdk-core-1.10.0.jar
-rw-rw-r-- 1 ec2-user ec2-user 2153751  4 août  16:32 aws-java-sdk-ec2-1.10.0.jar
-rw-rw-r-- 1 ec2-user ec2-user  258130  4 août  16:32 aws-java-sdk-kms-1.10.0.jar
-rw-rw-r-- 1 ec2-user ec2-user  563917  4 août  16:32 aws-java-sdk-s3-1.10.0.jar
-rw-rw-r-- 1 ec2-user ec2-user  232771  4 août  16:32 commons-codec-1.6.jar
-rw-rw-r-- 1 ec2-user ec2-user   62050  4 août  16:32 commons-logging-1.1.3.jar
-rw-rw-r-- 1 ec2-user ec2-user   44814  4 août  16:32 elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar
-rw-rw-r-- 1 ec2-user ec2-user  592008  4 août  16:32 httpclient-4.3.6.jar
-rw-rw-r-- 1 ec2-user ec2-user  282793  4 août  16:32 httpcore-4.3.3.jar
-rw-rw-r-- 1 ec2-user ec2-user   39815  4 août  16:32 jackson-annotations-2.5.0.jar
-rw-rw-r-- 1 ec2-user ec2-user 1143162  4 août  16:32 jackson-databind-2.5.3.jar
-rw-rw-r-- 1 ec2-user ec2-user    2253  4 août  16:32 plugin-descriptor.properties

And the cloud jar file contains the class we are looking for: org/elasticsearch/discovery/ec2/Ec2DiscoveryModule.class

[ec2-user@ip-10-0-0-113 cloud-aws]$ unzip -l elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar 
Archive:  elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  08-04-2015 17:40   META-INF/
      525  08-04-2015 17:40   META-INF/MANIFEST.MF
        0  08-04-2015 17:40   org/
        0  08-04-2015 17:40   org/elasticsearch/
        0  08-04-2015 17:40   org/elasticsearch/cloud/
        0  08-04-2015 17:40   org/elasticsearch/cloud/aws/
        0  08-04-2015 17:40   org/elasticsearch/cloud/aws/blobstore/
        0  08-04-2015 17:40   org/elasticsearch/cloud/aws/network/
        0  08-04-2015 17:40   org/elasticsearch/cloud/aws/node/
        0  08-04-2015 17:40   org/elasticsearch/discovery/
        0  08-04-2015 17:40   org/elasticsearch/discovery/ec2/
        0  08-04-2015 17:40   org/elasticsearch/plugin/
        0  08-04-2015 17:40   org/elasticsearch/plugin/cloud/
        0  08-04-2015 17:40   org/elasticsearch/plugin/cloud/aws/
        0  08-04-2015 17:40   org/elasticsearch/repositories/
        0  08-04-2015 17:40   org/elasticsearch/repositories/s3/
     8232  08-04-2015 17:40   org/elasticsearch/cloud/aws/AwsEc2Service.class
     1641  08-04-2015 17:40   org/elasticsearch/cloud/aws/AwsModule.class
      689  08-04-2015 17:40   org/elasticsearch/cloud/aws/AwsS3Service.class
     1232  08-04-2015 17:40   org/elasticsearch/cloud/aws/AwsSigner.class
    11319  08-04-2015 17:40   org/elasticsearch/cloud/aws/blobstore/DefaultS3OutputStream.class
     7517  08-04-2015 17:40   org/elasticsearch/cloud/aws/blobstore/S3BlobContainer.class
     6371  08-04-2015 17:40   org/elasticsearch/cloud/aws/blobstore/S3BlobStore.class
     3246  08-04-2015 17:40   org/elasticsearch/cloud/aws/blobstore/S3OutputStream.class
     8992  08-04-2015 17:40   org/elasticsearch/cloud/aws/InternalAwsS3Service.class
     2076  08-04-2015 17:40   org/elasticsearch/cloud/aws/network/Ec2NameResolver$Ec2HostnameType.class
     4004  08-04-2015 17:40   org/elasticsearch/cloud/aws/network/Ec2NameResolver.class
     3668  08-04-2015 17:40   org/elasticsearch/cloud/aws/node/Ec2CustomNodeAttributes.class
     1139  08-04-2015 17:40   org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider$1.class
     1472  08-04-2015 17:40   org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider$HostType.class
     9807  08-04-2015 17:40   org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.class
     2317  08-04-2015 17:40   org/elasticsearch/discovery/ec2/Ec2Discovery.class
     1635  08-04-2015 17:40   org/elasticsearch/discovery/ec2/Ec2DiscoveryModule.class
     2631  08-04-2015 17:40   org/elasticsearch/plugin/cloud/aws/CloudAwsPlugin.class
     6781  08-04-2015 17:40   org/elasticsearch/repositories/s3/S3Repository.class
     1099  08-04-2015 17:40   org/elasticsearch/repositories/s3/S3RepositoryModule.class
        0  08-04-2015 17:40   META-INF/maven/
        0  08-04-2015 17:40   META-INF/maven/org.elasticsearch.plugin/
        0  08-04-2015 17:40   META-INF/maven/org.elasticsearch.plugin/elasticsearch-cloud-aws/
     2172  08-04-2015 16:39   META-INF/maven/org.elasticsearch.plugin/elasticsearch-cloud-aws/pom.xml
      142  08-04-2015 17:40   META-INF/maven/org.elasticsearch.plugin/elasticsearch-cloud-aws/pom.properties
---------                     -------
    88707                     41 files
@dadoonet dadoonet added >bug blocker :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure :Plugin Cloud AWS labels Aug 4, 2015
@rmuir
Copy link
Contributor

rmuir commented Aug 4, 2015

We need tests that test this stuff... that needs to block any bugfix here.

I know if someone can fix it, they will feel pressure to push fix without tests. But we cannot develop software this way anymore!

@dadoonet
Copy link
Member Author

dadoonet commented Aug 4, 2015

So I tried to run Ec2DiscoveryITest from my IDE using options:

-Dtests.thirdparty=true -Dtests.config=/Users/dpilato/Documents/Elasticsearch/work/aws/elasticsearch.yml -Des.logger.level=DEBUG

And it went well. My config is:

cloud:
  aws:
    access_key: "KEY"
    secret_key: "SECRET"

discovery.type: ec2

Same when running from command line with mvn test, I don't get this error.

So I get only this when deploying the plugin in elasticsearch and trying to load AWS module.

@rmuir
Copy link
Contributor

rmuir commented Aug 4, 2015

ok, I think one thing we can do is cutover the thirdparty tests to run in the integration test phase so they run realistically? they are definitely integration tests. then they can be run in jenkins and we have more coverage for the discovery plugins: currently their integration tests only check that they were installed.

@dadoonet
Copy link
Member Author

dadoonet commented Aug 4, 2015

So the problem is coming from the changes in ClassLoading.

The Discovery Module is started within a Node using the Node settings (and the Node classloader aka elasticsearch core classloader).
When the plugin is loaded, it's now loaded within a child classloader.

Discovery then tries to call something like Classes.loadClass(settings.getClassLoader(), ...);.

settings.getClassLoader() is actually elasticsearch core Classloader, not the plugin one.

So we need somehow to tell the discovery module that he should use the plugin Classloader if any.

I'm not sure yet how to do that.

I pushed some changes for now in my branch https://github.com/dadoonet/elasticsearch/tree/plugins/discovery but it does not fix the issue. It only falls back to the default discovery instead of failling elasticsearch to start.

@dadoonet
Copy link
Member Author

dadoonet commented Aug 4, 2015

To reproduce the issue:

Build aws or azure plugin:

cd plugins/cloud-aws
mvn clean install -DskipTests

Build core:

cd core
mvn clean install -DskipTests

Build a distribution:

cd distribution/tar
mvn clean install -DskipTests
cd target/releases
tar xzf elasticsearch-2.0.0-SNAPSHOT.tar.gz
cd elasticsearch-2.0.0-SNAPSHOT
bin/plugin install cloud-aws --url file:../../../../../plugins/cloud-aws/target/releases/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip

Change config/elasticsearch.yml to:

discovery.type: ec2

Then start elasticsearch.

If you want to iterate other the code, just in core run mvn package and then copy the elasticsearch jar file from target to distribution/tar/target/releases/elasticsearch-2.0.0-SNAPSHOT/lib.

This was referenced Aug 4, 2015
Closed
Merged
rjernst added a commit to rjernst/elasticsearch that referenced this issue Aug 10, 2015
@rjernst
This method on settings loaded a class, based on a setting value, using
40f119d 
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes elastic#12643
closes elastic#12656
@clintongormley clintongormley added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Cloud AWS labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocker >bug :Distributed Coordination/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove Settings.getAsClass rjernst/elasticsearch
3 participants
@clintongormley @dadoonet @rmuir