Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

may relate to library mysql-binlog-connector-java.jar with issue to connect azure mysql(ssl enabled) #1295

Closed
plutoid opened this issue Jul 19, 2019 · 3 comments
Closed

may relate to library mysql-binlog-connector-java.jar with issue to connect azure mysql(ssl enabled) #1295

plutoid opened this issue Jul 19, 2019 · 3 comments

Comments

@plutoid
Copy link

plutoid commented Jul 19, 2019

hi,

looks I have issue to connect to mysql binlog again, this maybe also relate to history issue #1239, but not the same cause, since the mysql have enabled the ssl requirement, I'm not very sure about it, kindly help this.

my packages version:

Azure mysql 5.6 enabled ssl requirement
maxwell-1.22.3 - mysql-binlog-connector-java-0.20.jar
also tried:mysql-binlog-connector-java-0.20.1.jar

have tried ssl likely parameters:

--ssl='VERIFY_IDENTITY'  --replication_ssl='REQUIRED'   --schema_ssl='REQUIRED'   --jdbc_options='useSSL=true'

connection terminated by mysql server side as this tcpdump capture screenshot:
image

error log:

Using kafka version: 1.0.0
10:54:47,806 WARN  MaxwellMetrics - Metrics will not be exposed: metricsReportingType not configured.
10:54:48,749 INFO  Maxwell - Maxwell v1.22.3 is booting (StdoutProducer), starting at Position[BinlogPosition[mysql-bin.000001:353956],
 lastHeartbeat=0]
10:54:48,923 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[a97ae581-a6c7-11e9-a266-05229ba56
611:1-48], lastHeartbeat=0])
10:54:49,185 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin.000001:353956
10:54:49,315 INFO  TaskManager - Stopping 3 tasks
10:54:49,315 ERROR TaskManager - cause:
java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138) ~[?:1.8.0_192]
        at com.github.shyiko.mysql.binlog.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:68) ~[mysql-binlog-connector-java-0
.20.0.jar:0.20.0]
        at java.io.OutputStream.write(OutputStream.java:116) ~[?:1.8.0_192]
        at com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.write(PacketChannel.java:68) ~[mysql-binlog-connector-java-0.2
0.0.jar:0.20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient.authenticate(BinaryLogClient.java:718) ~[mysql-binlog-connector-java-0.20.0.j
ar:0.20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:515) ~[mysql-binlog-connector-java-0.20.0.jar:0.
20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857) ~[mysql-binlog-connector-java-0.20.0.jar:0.20
.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
10:54:49,320 INFO  TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@38522978
10:54:49,320 INFO  StoppableTaskState - com.zendesk.maxwell.schema.PositionStoreThread requestStop() called (in state: RUNNING)
10:54:49,320 INFO  TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@5728cc13
10:54:49,320 INFO  StoppableTaskState - com.zendesk.maxwell.bootstrap.BootstrapController requestStop() called (in state: RUNNING)
10:54:49,320 INFO  TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@411d7aaa
10:54:49,320 INFO  StoppableTaskState - com.zendesk.maxwell.replication.BinlogConnectorReplicator requestStop() called (in state: RUNNI
NG)
java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138)
        at com.github.shyiko.mysql.binlog.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:68)
        at java.io.OutputStream.write(OutputStream.java:116)
        at com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.write(PacketChannel.java:68)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.authenticate(BinaryLogClient.java:718)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:515)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
        at java.lang.Thread.run(Thread.java:748)
10:54:50,652 ERROR MaxwellContext - Exception occurred during shutdown:
java.util.concurrent.TimeoutException: Timed out trying waiting for com.zendesk.maxwell.replication.BinlogConnectorReplicator process t
o stop after 1000ms.
        at com.zendesk.maxwell.util.StoppableTaskState.awaitStop(StoppableTaskState.java:46) ~[maxwell-1.22.3.jar:1.22.3]
        at com.zendesk.maxwell.util.RunLoopProcess.awaitStop(RunLoopProcess.java:25) ~[maxwell-1.22.3.jar:1.22.3]
        at com.zendesk.maxwell.util.TaskManager.stop(TaskManager.java:53) ~[maxwell-1.22.3.jar:1.22.3]
        at com.zendesk.maxwell.MaxwellContext.shutdown(MaxwellContext.java:165) [maxwell-1.22.3.jar:1.22.3]
        at com.zendesk.maxwell.MaxwellContext.access$000(MaxwellContext.java:30) [maxwell-1.22.3.jar:1.22.3]
        at com.zendesk.maxwell.MaxwellContext$1$1.run(MaxwellContext.java:188) [maxwell-1.22.3.jar:1.22.3]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
10:54:50,653 ERROR MaxwellContext - Shutdown stalled - forcefully killing maxwell process
10:54:50,653 ERROR MaxwellContext - Termination reason:
java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138) ~[?:1.8.0_192]
        at com.github.shyiko.mysql.binlog.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:68) ~[mysql-binlog-connector-java-0
.20.0.jar:0.20.0]
        at java.io.OutputStream.write(OutputStream.java:116) ~[?:1.8.0_192]
        at com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.write(PacketChannel.java:68) ~[mysql-binlog-connector-java-0.2
0.0.jar:0.20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient.authenticate(BinaryLogClient.java:718) ~[mysql-binlog-connector-java-0.20.0.j
ar:0.20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:515) ~[mysql-binlog-connector-java-0.20.0.jar:0.
20.0]
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857) ~[mysql-binlog-connector-java-0.20.0.jar:0.20
.0]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]

Regards, Hua
@osheroff
Copy link
Collaborator

thanks for the bug report, Hua.

After we fixed the initial problems in Azure someone came along and fixed some more in binlog-connector 0.20.1. I think if we upgrade to that version your problem will go away.

@plutoid
Copy link
Author

plutoid commented Jul 19, 2019

hi @osheroff ,

thanks a lot for your quick reply, I did try the latest version ‎ 0.20.1 too, but no luck yet. :(

download this and replace the one in maxwell package:
https://search.maven.org/remotecontent?filepath=com/github/shyiko/mysql-binlog-connector-java/0.20.1/mysql-binlog-connector-java-0.20.1.jar

/opt/maxwell-1.22.3 ls -l /opt/maxwell-1.22.3/lib/mysql*
-rw-r--r--    1 root     root        173228 Jul 19 17:49 /opt/maxwell-1.22.3/lib/mysql-binlog-connector-java-0.20.1.jar
-rw-r--r--    1 501      dialout    1007502 Jun 20 18:11 /opt/maxwell-1.22.3/lib/mysql-connector-java-5.1.47.jar

Using kafka version: 1.0.0
17:49:55,533 WARN  MaxwellMetrics - Metrics will not be exposed: metricsReportingType not configured.
17:49:56,452 INFO  Maxwell - Maxwell v1.22.3 is booting (StdoutProducer), starting at Position[BinlogPosition[mysql-bin.000001:353956],
 lastHeartbeat=0]
17:49:56,690 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Position[BinlogPosition[a97ae581-a6c7-11e9-a266-05229ba56
611:1-48], lastHeartbeat=0])
17:49:56,943 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: mysql-bin.000001:353956
17:49:57,080 INFO  TaskManager - Stopping 3 tasks
17:49:57,080 ERROR TaskManager - cause:
java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_192]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431) ~[?:1.8.0_192]
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879) ~[?:1.8.0_192]
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:1.8.0_192]
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138) ~[?:1.8.0_192]
        at com.github.shyiko.mysql.binlog.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:68) ~[mysql-binlog-connector-java-0
.20.1.jar:0.20.1]  <============here is the version 0.20.1

@osheroff
Copy link
Collaborator

osheroff commented May 2, 2020

hi @plutoid I think this will be fixed with an upgrade to the latest maxwell. thx!

@osheroff osheroff closed this as completed May 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@osheroff @plutoid