Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault when many MQTTAsync clients connect to broker #234

Closed
6gt opened this issue Mar 7, 2017 · 5 comments
Closed

Segmentation fault when many MQTTAsync clients connect to broker #234

6gt opened this issue Mar 7, 2017 · 5 comments
Assignees
@icraggs
Labels
bug duplicate fix added
Milestone
1.2

Comments

@6gt
Copy link

6gt commented Mar 7, 2017

The test program create 2000 MQTTAsync clients and connect them to broker server.
After the 1089th client created, the program received SIGSEGV when connect it to broker server.
The trace log and program stack backtrace are shown below, the system open files limit has been set as 65535, is there any other limit cause this problem?

20170306 133407.998 Connecting to serverURI 192.168.0.5:1883 with MQTT version 4
20170306 133407.998 (3360278272) (2)> MQTTProtocol_connect:94
20170306 133407.998 (3360278272) (3)> MQTTProtocol_addressPort:51
20170306 133407.998 (3360278272) (4)> MQTTStrncpy:736
20170306 133407.998 the src string is truncated
20170306 133407.998 (3360278272) (4)< MQTTStrncpy:746
20170306 133407.998 (3360278272) (3)< MQTTProtocol_addressPort:72
20170306 133407.998 (3360278272) (3)> Socket_new:603
20170306 133407.998 New socket 1088 for 192.168.0.5, port 1883
20170306 133407.998 (3360278272) (4)> Socket_addSocket:163
20170306 133407.998 (3360278272) (5)> Socket_setnonblocking:73
20170306 133407.998 (3360278272) (5)< Socket_setnonblocking:78 (0)
20170306 133407.998 (3360278272) (4)< Socket_addSocket:176 (0)
20170306 133407.998 (3360278272) (4)> Socket_error:95
20170306 133407.998 (3360278272) (4)< Socket_error:104 (115)
20170306 133407.998 Connect pending
20170306 133407.998 (3360278272) (3)< Socket_new:687 (115)
20170306 133407.998 (3360278272) (2)< MQTTProtocol_connect:129 (115)
20170306 133407.998 (3360278272) (1)< MQTTAsync_processCommand:1314 (1)
20170306 133407.998 (3360278272) (1)> Thread_wait_cond:402
20170306 133407.998 (2521413376) (2)> MQTTAsync_connecting:2710
20170306 133407.998 (2521413376) (3)> MQTTPacket_send_connect:50
20170306 133407.998 (2521413376) (4)> MQTTPacket_send:180
20170306 133407.998 (2521413376) (5)> MQTTPacket_encode:269
20170306 133407.998 (2521413376) (5)< MQTTPacket_encode:279 (1)
20170306 133407.998 (2521413376) (5)> Socket_putdatas:445
20170306 133407.998 (2521413376) (6)> Socket_writev:400
20170306 133407.998 (2521413376) (6)< Socket_writev:421 (89)
20170306 133407.998 (2521413376) (5)< Socket_putdatas:487 (0)
20170306 133407.998 (2521413376) (4)< MQTTPacket_send:207 (0)
20170306 133407.998 697 prjcdrcnfu65064300030190 -> CONNECT cleansession: 1 (0)
20170306 133407.998 (2521413376) (3)< MQTTPacket_send_connect:108 (0)
20170306 133407.998 (2521413376) (2)< MQTTAsync_connecting:2818 (0)
20170306 133407.998 (2521413376) (2)> MQTTAsync_retry:2692
20170306 133407.998 (2521413376) (3)> MQTTProtocol_retry:629
20170306 133407.998 (2521413376) (3)< MQTTProtocol_retry:648
20170306 133407.998 (2521413376) (2)< MQTTAsync_retry:2702
20170306 133407.998 (2521413376) (1)< MQTTAsync_cycle:2968 (0)
20170306 133407.998 (3455940672) (0)< MQTTAsync_setCallbacks:1968 (0)
20170306 133407.998 (3455940672) (0)> MQTTAsync_connect:2153
20170306 133407.998 (3455940672) (1)> MQTTStrncpy:736
20170306 133407.998 (3455940672) (1)< MQTTStrncpy:746
20170306 133407.998 (3455940672) (1)> MQTTStrncpy:736
20170306 133407.998 (3455940672) (1)< MQTTStrncpy:746
20170306 133407.998 (3455940672) (1)> MQTTAsync_addCommand:805
20170306 133407.998 (3455940672) (1)< MQTTAsync_addCommand:838 (0)
20170306 133407.998 (3455940672) (0)< MQTTAsync_connect:2313 (0)
20170306 133407.998 (2521413376) (1)> MQTTAsync_cycle:2829
20170306 133407.998 (2521413376) (2)> Socket_getReadySocket:217
20170306 133407.998 (2521413376) (3)> isReady:193
20170306 133407.998 (2521413376) (3)< isReady:198 (1)
20170306 133407.998 (2521413376) (2)< Socket_getReadySocket:283 (698)
20170306 133407.998 (3360278272) (1)< Thread_wait_cond:416 (0)
20170306 133407.998 (3360278272) (1)> MQTTAsync_checkTimeouts:1325
20170306 133407.998 (3360278272) (1)< MQTTAsync_checkTimeouts:1420
20170306 133407.998 (3360278272) (1)> MQTTAsync_processCommand:1065
20170306 133407.998 (3455940672) (0)> MQTTAsync_createWithOptions:408
20170306 133407.998 (2521413376) (2)> MQTTAsync_connecting:2710
20170306 133407.998 (2521413376) (3)> MQTTPacket_send_connect:50
20170306 133407.998 (2521413376) (4)> MQTTPacket_send:180
20170306 133407.998 (2521413376) (5)> MQTTPacket_encode:269
20170306 133407.999 (2521413376) (5)< MQTTPacket_encode:279 (1)
20170306 133407.999 (2521413376) (5)> Socket_putdatas:445
20170306 133407.999 (2521413376) (6)> Socket_writev:400
20170306 133407.999 (2521413376) (6)< Socket_writev:421 (89)
20170306 133407.999 (2521413376) (5)< Socket_putdatas:487 (0)
20170306 133407.999 (2521413376) (4)< MQTTPacket_send:207 (0)
20170306 133407.999 698 prjcdrcnfu65064300030191 -> CONNECT cleansession: 1 (0)
20170306 133407.999 (2521413376) (3)< MQTTPacket_send_connect:108 (0)
20170306 133407.999 (2521413376) (2)< MQTTAsync_connecting:2818 (0)
20170306 133407.999 (2521413376) (2)> MQTTAsync_retry:2692
20170306 133407.999 (2521413376) (3)> MQTTProtocol_retry:629
20170306 133407.999 (2521413376) (3)< MQTTProtocol_retry:648
20170306 133407.999 (2521413376) (2)< MQTTAsync_retry:2702
20170306 133407.999 (2521413376) (1)< MQTTAsync_cycle:2968 (0)
20170306 133407.999 Connecting to serverURI 192.168.0.5:1883 with MQTT version 4
20170306 133407.999 (3360278272) (2)> MQTTProtocol_connect:94
20170306 133407.999 (3360278272) (3)> MQTTProtocol_addressPort:51
20170306 133407.999 (3360278272) (4)> MQTTStrncpy:736
20170306 133407.999 the src string is truncated
20170306 133407.999 (3360278272) (4)< MQTTStrncpy:746
20170306 133407.999 (3360278272) (3)< MQTTProtocol_addressPort:72
20170306 133407.999 (3360278272) (3)> Socket_new:603
20170306 133407.999 New socket 1089 for 192.168.0.5, port 1883
20170306 133407.999 (3360278272) (4)> Socket_addSocket:163

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc849c700 (LWP 6495)]
ListFindItem (aList=0x2f77fc5, content=content@entry=0x7fffc849bc8c, callback=0x7fffcdb2f795 ) at src/LinkedList.c:155
155 src/LinkedList.c: No such file or directory.
(gdb) bt
#0 ListFindItem (aList=0x2f77fc5, content=content@entry=0x7fffc849bc8c, callback=0x7fffcdb2f795 ) at src/LinkedList.c:155
#1 0x00007fffcdb3a9bb in Socket_addSocket (newSd=1089) at src/Socket.c:164
#2 0x00007fffcdb3b3c1 in Socket_new (addr=addr@entry=0x7fff60051344 "192.168.0.5", port=1883, sock=sock@entry=0x32cd974) at src/Socket.c:664
#3 0x00007fffcdb30ce1 in MQTTProtocol_connect (ip_address=ip_address@entry=0x32cd704 "192.168.0.5:1883", aClient=0x32cd954, MQTTVersion=4) at src/MQTTProtocolOut.c:98
#4 0x00007fffcdb338dc in MQTTAsync_processCommand () at src/MQTTAsync.c:1159
#5 0x00007fffcdb33d6e in MQTTAsync_sendThread (n=) at src/MQTTAsync.c:1437
#6 0x00007fffcca62df3 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fffcc7903dd in clone () from /lib64/libc.so.6
(gdb)

[root@localhost ~]# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30197
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 819200
cpu time (seconds, -t) unlimited
max user processes (-u) 30197
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
@Jarkre
Copy link

Jarkre commented Mar 10, 2017

Try with these settings when creating clients:

sendWhileDisconnected = 0
maxBufferedMessages = 1

It this works then gradually increase maxBufferedMessages (10 -> 30 etc.)

@icraggs icraggs self-assigned this Apr 7, 2017
@icraggs icraggs added the bug label Apr 7, 2017
@icraggs icraggs added this to the 1.2 milestone Apr 7, 2017
@icraggs
Copy link
Contributor

icraggs commented Apr 7, 2017

I think this is caused by #186, which I will look to fix.

@icraggs
Copy link
Contributor

icraggs commented Apr 24, 2017

FD_SETSIZE is limited to 1024 on Linux.

@icraggs
Copy link
Contributor

icraggs commented Apr 30, 2017

I've added a fix to the develop branch to return an error on connect when FD_SETSIZE is exceeded.

@icraggs
Copy link
Contributor

icraggs commented Jun 28, 2017

Will be in 1.2

@icraggs icraggs closed this as completed Jun 28, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@icraggs icraggs

Labels
bug duplicate fix added
Projects
None yet
Milestone
1.2
Development

No branches or pull requests
3 participants
@6gt @Jarkre @icraggs