short read when reading frame size

[mileschao]

Hi.
While using pulsar client, I got

level=info message "Error reading from connection" error="Short read when reading frame size"

Question is:
what does this log mean?
what can I do for it, or, How could I deal with it?

Thx for your help.
@sijie @wolfstudy

[wolfstudy]

Thanks @mileschao for the feedback, the problem seems to be an error reading the message format. Can you provide more information?

[zzzming]

similar to this one #145 ? It's supposed to be fixed. Which commit are the client on @mileschao

[mileschao]

Thanks @mileschao for the feedback, the problem seems to be an error reading the message format. Can you provide more information?

It is a weird problem.
I create a simple client (with the latest version on master branch), and publish messages continually to the pulsar server for a long time.
sometimes, the Short read when reading frame size error level log will appear.

I think it is hard to reproduce.

My real question is:
Is there any way to deal with it?
cause it just break the for loop and log it in the code.
and I can not find any way to deal with this error, for example, reconnect, or recreate.
Thx for your time. :-)

[mileschao]

similar to this one #145 ? It's supposed to be fixed. Which commit are the client on @mileschao

Em.
It may be a useful hint, but I can not figure it out.

I've got the same error information, but, no pulsar proxy here.

I just create a simple client, and continually publish messages to the pulsar broker.
Thx for your time. :-)

[sijie]

@mileschao do you encounter any issue when producing? or is the logging just an info logging?

[mileschao]

@mileschao do you encounter any issue when producing? or is the logging just an info logging?

@sijie Thx for your attention.

Yes, I can reproduce it frequently.
and. the log is:

level=info message "Error reading from connection" error="Short read when reading frame size"

The real problem is, there is nothing I can do for this.
custom handler for this problem is wanted. :-)

[sijie]

@mileschao thank you for your reply. Does it impact your application? Or just an error message?

[mileschao]

@mileschao thank you for your reply. Does it impact your application? Or just an error message?

producer will stop send message to pulsar cluster.
and the for loop break.
readSingleCommand

for {
		cmd, headersAndPayload, err := r.readSingleCommand()
		if err != nil {
			r.cnx.log.WithError(err).Info("Error reading from connection")
			r.cnx.TriggerClose()
			break
		}

		// Process
		var payloadLen uint32
		if headersAndPayload != nil {
			payloadLen = headersAndPayload.ReadableBytes()
		}
		r.cnx.log.Debug("Got command! ", cmd, " with payload size: ", payloadLen)
		r.cnx.receivedCommand(cmd, headersAndPayload)
	}

[wolfstudy]

@mileschao The #237 has been merged, and i will close the issue. If you reproduce the problem again, or if there is any update, please reopen it, thanks.

[mileschao]

time="2020-05-25T14:11:45+08:00" level=warning msg="Detected stale connection to broker" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650"
time="2020-05-25T14:11:45+08:00" level=warning msg="Detected stale connection to broker" local_addr="10.187.98.151:41310" remote_addr="pulsar://10.187.128.66:6650"
time="2020-05-25T14:11:45+08:00" level=warning msg="Failed to write on connection" error="write tcp 10.187.98.151:54472->10.187.128.67:6650: write: broken pipe" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650"
time="2020-05-25T14:11:45+08:00" level=info msg="Error reading from connection" error="Short read when reading frame size: EOF" local_addr="10.187.98.151:54472" remote_addr="pulsar://10.187.128.67:6650"
time="2020-05-25T14:11:45+08:00" level=warning msg="Failed to write on connection" error="write tcp 10.187.98.151:41310->10.187.128.66:6650: write: brokenpipe" local_addr="10.187.98.151:41310" remote_addr="pulsar://10.187.128.66:6650"

@wolfstudy
I got log above by PR

Short read when reading frame size: EOF

How could I deal with it?

[jackwang123]

i encounter the same problem, have you resolved this problem?

[kwenzh]

Thanks @mileschao for the feedback, the problem seems to be an error reading the message format. Can you provide more information?

It is a weird problem. I create a simple client (with the latest version on master branch), and publish messages continually to the pulsar server for a long time. sometimes, the Short read when reading frame size error level log will appear.

I think it is hard to reproduce.

My real question is: Is there any way to deal with it? cause it just break the for loop and log it in the code. and I can not find any way to deal with this error, for example, reconnect, or recreate. Thx for your time. :-)

https://github.com/apache/pulsar-client-go/blob/master/pulsar/internal/connection_reader.go#L51
in this run exception , this gorutine will break and exit for. so never retry conn?