register_recv_callback - VMA_PACKET_HOLD segmentation fault

[rmcginnisaz]

I am attempting to develop and application which utilizes the 'register_recv_callback' API to setup a callback to receive data from multiple UDP sockets. This application is receiving data from 100-150 multicast sockets and processing the zero-copy packets within a callback function that was registered using 'register_recv_callback'. The application is receiving approximately 980,000 packets/sec from the multicast sockets and can run for hours at a time.

In the my callback function if I simply drop the packet (return VMA_PACKET_DROP) the application will work and never crash. When I modify the callback to return VMA_PACKET_HOLD and process/free the packet from another thread I see infrequent almost random exceptions and/or segmentation faults.

During long term testing using OFED 4.0-1.0.1.0 (VMA 8.2.8 included) I've seen the following errors:

VMA 8.2.8 - C++ Exception:
03-21-2017 23:47:51.558597 INFO: Pkts = 976557.208093, Drops=0, PDiffs=0
VMA INFO: bpool[0x806330]:568:buffersPanic() no circle was found in buffer_pool
VMA ERROR: 0 /lib64/libvma.so.8(_ZN11buffer_pool12buffersPanicEv+0x77) [0x7f4c7fc3d1c7]
VMA ERROR: 1 /lib64/libvma.so.8(_ZN11buffer_pool11put_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x14e) [0x7f4c7fc3d51e]
VMA ERROR: 2 /lib64/libvma.so.8(_ZN11buffer_pool23put_buffers_thread_safeEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEEm+0x2f) [0x7f4c7fc3d55f]
VMA ERROR: 3 /lib64/libvma.so.8(_ZN6cq_mgr20return_extra_buffersEv+0x45) [0x7f4c7fc3f7e5]
VMA ERROR: 4 /lib64/libvma.so.8(_ZN6cq_mgr20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x78) [0x7f4c7fc41088]
VMA ERROR: 5 /lib64/libvma.so.8(_ZN11ring_simple20reclaim_recv_buffersEP10vma_list_tI14mem_buf_desc_tXadL_ZNS1_18buffer_node_offsetEvEEE+0x3f) [0x7f4c7fc5aaff]
VMA ERROR: 6 /lib64/libvma.so.8(_ZN12sockinfo_udp12reuse_bufferEP14mem_buf_desc_t+0x11f) [0x7f4c7fcaa32f]
VMA ERROR: 7 /lib64/libvma.so.8(_ZN12sockinfo_udp12free_packetsEP12vma_packet_tm+0x6a) [0x7f4c7fca944a]
VMA ERROR: 8 ./tx2test5() [0x405316]
VMA ERROR: 9 /lib64/libpthread.so.0(+0x7dc5) [0x7f4c7f9d2dc5]
VMA ERROR: 10 /lib64/libc.so.6(clone+0x6d) [0x7f4c7e8021cd]
VMA PANIC: bpool[0x806330]:581:buffersPanic() m_n_buffers(200001) > m_n_buffers_created(200000)
terminate called without an active exception

VMA 8.2.8 - Segmentation Fault 1:
03-22-2017 16:30:36.548321 INFO: Pkts = 976554.977319, Drops=0, PDiffs=0
VMA ERROR: vma_list_t.push_back() - buff is already a member in a list.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault

VMA 8.2.8 - Segmentation Fault 2:
03-22-2017 20:58:18.255651 INFO: Pkts = 976546.831317, Over=0, Drops=0, PDiffs=0
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault

Noticing there had been changes made to many of the files since the VMA 8.2.8 release I downloaded and compiled VMA 8.2.10 and re-tested with updated VMA libraries. The likelyhood of occurance seems to be lower but I can seemingly still receive segementation faults:

VMA 8.2.10 - Segmentation Fault:
03-23-2017 20:44:54.231603 INFO: Pkts = 976554.117604, Over=0, Drops=0, PDiffs=0
03-23-2017 20:45:54.231729 INFO: Pkts = 976544.334772, Over=0, Drops=0, PDiffs=0
VMA ERROR: vma_list_t.push_back() - buff is already a member in a list.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
< rows deleted >
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
VMA WARNING: vma_list_t.erase() got NULL object - ignoring.
Segmentation fault

[rmcginnisaz]

OS: RHEL 7.2 & CentOS 7.2
NIC: ConnectX3