ThirdParty server hangup problem with 7nodes example

[bigstar119]

follow the suggest on #954. i use the latest realease of qurom and tessera deploy quorum-examples with 7 nodes. but ThirdParty server 9081-9087 port hangup again. this is my deploy and test steps, maybe can help reappear the bug.

1、prepare host server, operate system version：CentOS Linux release 7.3.1611 x86_64
2、download geth_v2.4.0_linux_amd64.tar.gz, url:https://github.com/jpmorganchase/quorum/releases
3、download tessera-app-0.10.2-app.jar,url:https://oss.sonatype.org/service/local/repositories/releases/content/com/jpmorgan/quorum/tessera-app/0.10.2/tessera-app-0.10.2-app.jar
4、download quorum-examples-master.zip,url: https://codeload.github.com/jpmorganchase/quorum-examples/zip/master
5、mkdir -p /home/cpap/app/blockchain/lib
cp tessera-app-0.10.2-app.jar /home/cpap/app/blockchain/lib
cp geth_v2.4.0_linux_amd64.tar.gz /home/cpap/app/blockchain/lib
cd /home/cpap/app/blockchain/lib
tar -xvf geth_v2.4.0_linux_amd64.tar.gz
cp geth /usr/local/bin
chmod 777 /usr/local/bin/geth
[root@hadoop-1 lib] geth --version
Incorrect Usage. flag provided but not defined: -version
NAME:
geth - the go-ethereum command line interface
Copyright 2013-2018 The go-ethereum Authors
USAGE:
geth [options] command [command options] [arguments...]
VERSION:
1.8.18-stable-20c95e5d
QUORUM OPTIONS:
--permissioned If enabled, the node will allow only a defined list of nodes to connect
...
6、mkdir -p /home/cpap/app/blockchain
cp quorum-examples-master.zip /home/cpap/app/blockchain
cd /home/cpap/app/blockchain
unzip quorum-examples-master.zip
cd quorum-examples-master/examples/7nodes/
vi ./tessera-init.sh
delete the cros config of P2P server in tessera-config-09-${i}.json,because tessera-app-0.10.2 only can config cros on the ThirdParty Server

7、start 7nodes
./raft-init.sh
./raft-start.sh tessera --tesseraOptions "--tesseraJar /home/cpap/app/blockchain/lib/tessera-app-0.10.2-app.jar"
...
All nodes configured. See 'qdata/logs' for logs, and run e.g. 'geth attach qdata/dd1/geth.ipc' to attach to the first Geth node.
To test sending a private transaction from Node 1 to Node 7, run './runscript.sh private-contract.js'

8、test 9081 port on 7nodes, every 10 minutes

first try everything is ok
[root@hadoop-1 7nodes] curl -s http://localhost:9081/upcheck
curl -s http://localhost:9082/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9082/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9083/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9084/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9085/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9086/upcheck
I'm up![root@hadoop-1 7nodes] curl -s http://localhost:9087/upcheck
I'm up![root@hadoop-1 7nodes] date
2020-01-08 13:03:36 CST

the second try 9081-9087 hangup
[root@hadoop-1 7nodes]# curl -s http://localhost:9081/upcheck
curl -s http://localhost:9082/upcheck
curl -s http://localhost:9083/upcheck
curl -s http://localhost:9084/upcheck
curl -s http://localhost:9085/upcheck
curl -s http://localhost:9086/upcheck
curl -s http://localhost:9087/upcheck
^C
[root@hadoop-1 7nodes]# date
2020-01-08 13:15:17

2020-01-08 13:15:17 bug reappear !!!

[SatpalSandhu61]

Ok, I'll see if I can recreate using your config and steps.
FYI We're running on MacBook (Darwin Kernel Version 19.2.0, xnu-6153.61.1~20/RELEASE_X86_64) but we do have access to some other OS versions.

[SatpalSandhu61]

BTW when the problem occurs is the server still listening on port 9081 (netstat -an | grep 9081)?

[SatpalSandhu61]

Ok, so the problem doesn't occur on the Mac so I suspect it is OS related. I don't believe we have access to a CentOS machine, but I'll test on some other Linux versions to see if I can reproduce.

In the meantime, would you be able to run Tessera with debug logging enabled? You'll need to create the following logback.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'[\r\n]', ''}%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
    </root>

</configuration>

Run this using:

/raft-start.sh tessera --tesseraOptions "--tesseraJar /home/cpap/app/blockchain/lib/tessera-app-0.10.2-app.jar --jvmParams -Dlogback.configurationFile=/path/to/logback.xml"

[bigstar119]

@SatpalSandhu61 when the problem occurs server still listening on port 9081
netstat -an|grep 908
tcp6 0 0 :::9081 ::: LISTEN
tcp6 0 0 :::9082 ::: LISTEN
tcp6 0 0 :::9083 :::* LISTEN
tcp6 0 0 :::9084 :::* LISTEN
tcp6 0 0 :::9085 :::* LISTEN
tcp6 0 0 :::9086 :::* LISTEN
tcp6 0 0 :::9087 :::* LISTEN**

[bigstar119]

in china most software company used win10 operation system for develop
and use CentOS/redhat/ubuntu for product
and on win10 i did not reappear the bug.. so I suspect the problem is OS related too.

[bigstar119]

jetty-server.zip
@SatpalSandhu61 I'm pretty sure that the problem is on the jetty+glassfish`s jersey.. i take out the core code form tessera and made a single test. jetty-server.zip is my test maven project. package the project you will get jetty-server-1.0-SNAPSHOT.jar .
run the jar on centos7 like this:
java -jar jetty-server-1.0-SNAPSHOT.jar

every 10 minutes send a http request to the server
soon you will reapear the bug

[root@hadoop-1 ~]# date
2020- 01-09日 15:10:19 CST
[root@hadoop-1 ~]# curl -s localhost:9081/upcheck
I'm up![root@hadoop-1 ~]# curl -s localhost:9082/upcheck
I'm up![root@hadoop-1 ~]# date
2020-01-09 15:15:51 CST
[root@hadoop-1 ~]# curl -s localhost:9081/upcheck
curl -s localhost:9082/upcheck
^C
[root@hadoop-1 ~]# curl -s localhost:9081/upcheck
^C
[root@hadoop-1 ~]# netstat -an|grep 9081
tcp6 0 0 :::9081 :::* LISTEN
tcp6 0 0 ::1:39782 ::1:9081 FIN_WAIT2
tcp6 86 0 ::1:9081 ::1:39476 CLOSE_WAIT
tcp6 0 0 ::1:39476 ::1:9081 FIN_WAIT2
tcp6 86 0 ::1:9081 ::1:39782 CLOSE_WAIT

[bigstar119]

hi. i think i find the reason of the problem. i update jetty.version from 9.4.17.v20190418 to 9.4.25.v20191220 . repackage and deploy tessera.jar to myhost. everything goes fine!!
this is my pom.xml

    <jetty.version>9.4.25.v20191220</jetty.version>

i think the jetty.project fixed our problem between version 9.4.17.v20190418 and 9.4.25.v20191220

[SatpalSandhu61]

@bigstar119 many thanks for investigating and narrowing that down!
We'll take a look at updating the jetty version.

[SatpalSandhu61]

@bigstar119 when you built tessera with the upgraded jetty, did you run the acceptance tests?

I tried upgrading to 9.4.25.v20191220, but it causes the tests to fail - it looks like there is an issue with recent jetty versions which result in an incompatibility with jersey's jetty test container. See here:
eclipse-ee4j/jersey#4302

[melowe]

I've just put a PR up to bump the jetty version as you said you had done on your local build. #968 . It builds for us and our tests pass, but please feel free to validate that it addresses the original issue.

[bigstar119]

hi guys i download the upgrade_jetty_version branch ,recompile and depoly to my host, everythings goes fine. i think we fixed the bug . good work!!!