Subscription appears to stop and strange numbers in AeronStat.

[mikeb01]

Similar problem to before, but much rarer (1 server in 70) and with different aeron stat info. One of the subscriptions appears to not be receiving messages. In this case the Publication seems to be happily handling offered messages (no BACK_PRESSURE responses).

General Aeron stat values:

03:35:25 - Aeron Stat
=========================
  0:           59,329,600 - Bytes sent
  1:           60,361,216 - Bytes received
  2:                    0 - Failed offers to ReceiverProxy
  3:                    0 - Failed offers to SenderProxy
  4:                    0 - Failed offers to DriverConductorProxy
  5:               27,624 - NAKs sent
  6:               27,624 - NAKs received
  7:              870,770 - Status Messages sent
  8:              870,615 - Status Messages received
  9:            1,679,922 - Heartbeats sent
 10:            1,537,896 - Heartbeats received
 11:                9,847 - Retransmits sent
 12:                1,875 - Flow control under runs
 13:                    0 - Flow control over runs
 14:                    0 - Invalid packets
 15:                    0 - Errors
 16:                    0 - Short sends
 17:               35,886 - Client keep-alives
 18:                  437 - Sender flow control limits applied
 19:                    0 - Unblocked Publications
 20:                    0 - Unblocked Control Commands
 21:                    0 - Possible TTL Asymmetry
 22:                    1 - send-channel: aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1

And the values for the stream that has stopped:

999:            2,020,544 - pub-lmt: 876 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 
1050:            1,987,776 - snd-pos: 876 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 
1250:        1,191,440,576 - sub-pos: 64 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1251:        1,191,440,576 - sub-pos: 194 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1252:        1,191,440,576 - sub-pos: 280 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1253:        1,191,440,576 - sub-pos: 303 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1254:        1,191,440,576 - sub-pos: 328 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1255:        1,191,440,576 - sub-pos: 652 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1256:        1,191,440,576 - sub-pos: 704 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1257:        1,191,440,576 - sub-pos: 763 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1258:            1,987,776 - rcv-hwm: 936 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 
1259:        1,191,440,576 - rcv-pos: 936 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 
1333:        1,191,440,576 - sub-pos: 1161 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1357:        1,191,440,576 - sub-pos: 1191 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1612:        1,191,440,576 - sub-pos: 1212 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @0
1874:        1,191,440,576 - sub-pos: 1440 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @23,936
2110:        1,191,440,576 - sub-pos: 1628 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @144,192
2239:        1,191,440,576 - sub-pos: 1735 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @1,191,440,576
2348:        1,191,440,576 - sub-pos: 1836 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @1,191,440,576
2378:        1,191,440,576 - sub-pos: 1862 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @1,191,440,576
2397:        1,191,440,576 - sub-pos: 1896 -1175656989 -1762877036 aeron:udp?endpoint=239.193.0.1:10000|interface=172.29.1.1 @1,191,440,576

The weird part is the very large values for the subscriber position. That suggest I've shipped ~1.1 GB of messages, but that doesn't match up with the total bytes sent in the general aeron stats.

[mjpt777]

Do you have the log buffers for this?

[mikeb01]

Image, publication and cnc file attached aeron-data.zip.

[gierlachg]

@mikeb01 From what I can see subscriber log buffer is zeroed out. I was observing exactly same symptom (#268) and unless it's not a different issue the fix should be already in.

[mikeb01]

@grzegorz-gierlach I thought the issue might have the same cause. I'll pick up the head build and test again next week.

[mikeb01]

Closing this one. Started successfully with version 1.0.1. Think that #268 has fixed it.