Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build - Hudson build stuck waiting to connect to Elasticsearch node #829

Closed
stefanomorson opened this issue Sep 22, 2017 · 7 comments
Closed

Build - Hudson build stuck waiting to connect to Elasticsearch node #829

stefanomorson opened this issue Sep 22, 2017 · 7 comments
Assignees
@muros-ct
Labels
Bug This is a bug or an unexpected behaviour. Fix it! build Test Test related stuff. It's a dirty job, but someone needs to do that!

Comments

@stefanomorson
Copy link
Contributor

Here is the log from Hudson:

INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.eclipse.kapua.service.datastore.client.transport.EsClientTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.019 s - in org.eclipse.kapua.service.datastore.client.transport.EsClientTest
[INFO] Running org.eclipse.kapua.service.datastore.client.transport.EsTransportClientProviderTest
04:39:09.883 [main] DEBUG org.apache.commons.configuration.PropertiesConfiguration - Base path set to file:///jobs/genie.kapua/develop/workspace/service/datastore/client-transport/target/test-classes/
04:39:09.888 [main] DEBUG org.apache.commons.configuration.PropertiesConfiguration - FileName set to kapua-datastore-transport-client-setting.properties
04:39:09.888 [main] DEBUG org.apache.commons.configuration.PropertiesConfiguration - URL set to file:/jobs/genie.kapua/develop/workspace/service/datastore/client-transport/target/test-classes/kapua-datastore-transport-client-setting.properties
04:39:09.907 [main] DEBUG org.eclipse.kapua.commons.setting.AbstractKapuaSetting - Loaded configuration resource: 'kapua-datastore-transport-client-setting.properties'
04:39:10.158 [main] INFO org.eclipse.kapua.service.datastore.client.transport.EsTransportClientProvider - >>> Initializing ES transport client...
04:39:12.117 [main] INFO org.elasticsearch.plugins.PluginsService - no modules loaded
04:39:12.120 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
04:39:12.120 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
04:39:12.120 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
04:39:12.120 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
04:39:12.120 [main] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
04:39:12.223 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [force_merge], size [1], queue size [unbounded]
04:39:12.224 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_started], core [1], max [32], keep alive [5m]
04:39:12.224 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [listener], size [8], queue size [unbounded]
04:39:12.229 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [index], size [16], queue size [200]
04:39:12.229 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [refresh], core [1], max [8], keep alive [5m]
04:39:12.229 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [generic], core [4], max [128], keep alive [30s]
04:39:12.229 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [warmer], core [1], max [5], keep alive [5m]
04:39:12.230 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [search], size [25], queue size [1k]
04:39:12.230 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [flush], core [1], max [5], keep alive [5m]
04:39:12.230 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_store], core [1], max [32], keep alive [5m]
04:39:12.230 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [management], core [1], max [5], keep alive [5m]
04:39:12.230 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [get], size [16], queue size [1k]
04:39:12.231 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [bulk], size [16], queue size [200]
04:39:12.231 [main] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [snapshot], core [1], max [5], keep alive [5m]
04:39:12.237 [main] DEBUG org.elasticsearch.common.network.IfConfig - configuration:

lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
inet6 ::1 prefixlen:128 scope:host
UP LOOPBACK mtu:65536 index:1

eth0
inet 172.30.206.109 netmask:255.255.255.0 broadcast:172.30.206.255 scope:site
inet6 fe80::225:90ff:fefd:404 prefixlen:64 scope:link
hardware 00:25:90:FD:04:04
UP MULTICAST mtu:1500 index:2

eth1
inet 172.25.102.109 netmask:255.255.0.0 broadcast:172.25.255.255 scope:site
inet6 fe80::225:90ff:fefd:405 prefixlen:64 scope:link
hardware 00:25:90:FD:04:05
UP MULTICAST mtu:1500 index:3
@stefanomorson stefanomorson added Bug This is a bug or an unexpected behaviour. Fix it! build labels Sep 22, 2017
@stefanomorson stefanomorson changed the title Build - Hudson build stuck waiting to connect embedded Elastic node Build - Hudson build stuck waiting to connect to Elasticsearch node Sep 22, 2017
@muros-ct muros-ct modified the milestone: Kapua 1.0 Oct 17, 2017
@muros-ct
Copy link
Contributor

Moved to 1.0.0 as it has to be solved to provide successful Eclipse Hudson CI build / test.

@muros-ct muros-ct added the Test Test related stuff. It's a dirty job, but someone needs to do that! label Oct 17, 2017
muros-ct referenced this issue Oct 17, 2017
@muros-ct
Tests locking on connection_timeout ignored
3061a3f 
EsTransportClientProvider tests that test connection on non existent host are Ignored
for now, as they lock up Hudson CI environment.
This two tests that are ignored should be revisited and functionality tested.

This failure also brings connection timeout in question and should be tested too.

Signed-off-by: Uros Mesaric Kunst <uros.mesaric-kunst@comtrade.com>
@muros-ct
Copy link
Contributor

I commented out tests that timed-out on connection to non existent ES nodes.
But anoter issue with ES emerged.
While running tests in MessageStoreServiceTests EsEmbeddedEngine is started, and while it is started it hangs.

09:20:43.588 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - Starting Elasticsearch embedded node... (data directory: 'target/elasticsearch/data/8_y9ZNsqR6KnO56hrGElNQ') 09:20:43.591 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - Cluster name [kapua-datastore] 09:20:43.591 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - >>> Transport: host [127.0.0.1] - port [9300] 09:20:43.591 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - >>> Rest: host [127.0.0.1] - port [9200]
And it waits...

This brings into question whole embedded ES integration tests on Hudson environment, or more on specific Eclipse Hudson CI. Those tests run successfuly on Travis CI.

@muros-ct
Copy link
Contributor

This is from Eclipse Hudson CI.
It took 1.5 hour to build and run tests on datastore-internal.
This tests use embedded ES, but so do QA tests and those run normally.

[INFO] kapua-datastore-internal ........................... SUCCESS [ 01:35 h]

@muros-ct
Copy link
Contributor

This is from Eclipse Hudson raw output of datastore-internal tests.
It locks on embeded ES start. It might be due to start of ES in static block of test class.

12:07:48.459 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - Starting Elasticsearch embedded node... (data directory: 'target/elasticsearch/data/jn-3t48dQN2PDpOJkItBcA') 12:07:48.461 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - Cluster name [kapua-datastore] 12:07:48.461 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - >>> Transport: host [127.0.0.1] - port [9300] 12:07:48.461 [main] INFO o.e.k.s.d.c.e.EsEmbeddedEngine - >>> Rest: host [127.0.0.1] - port [9200] 13:33:11.380 [main] INFO org.elasticsearch.node.Node - initializing ...

@muros-ct
Copy link
Contributor

I moved static initializer of embedded ES into @before section of unit test MessageStoreServiceTest.

@muros-ct
Copy link
Contributor

muros-ct commented Oct 18, 2017 •
edited
Loading

Build on Eclipse CI is completing now under an hour and doesn't block on ES start.
But Sonar is still sometimes complaining that it is unstable.

@muros-ct
Copy link
Contributor

I will monitor the build and reopen the issue or create new one if problems emerge again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@muros-ct muros-ct

Labels
Bug This is a bug or an unexpected behaviour. Fix it! build Test Test related stuff. It's a dirty job, but someone needs to do that!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stefanomorson @muros-ct