NFS-Ganesha traffic is not getting offloaded

[syspro4]

Subject

NFS-Ganesha traffic is not getting offloaded

Issue type

• Bug report
• Feature request

Configuration:

• Product version : 9.4
• OS: Oracle Linux 8.3
• OFED: MLNX_OFED_LINUX-5.4-1.0.3.0
• Hardware: Mellanox Technologies MT27700 Family [ConnectX-4]

Actual behavior:

NFS-Ganesha traffic is not getting offloaded after running IOs from nfs client.
I created NFS-Ganesha configuration based on local xfs filesystem and used VFS FSAL to export the xfs. I started NFS-Ganesha with LD_PRELOAD=libvma.so & then did nfs mount from the host and did IOs but in vma_stats I did not see any offloaded counters. I saw Timeout values.

Following are the details:

NFS-Ganesha config:
$ cat /etc/ganesha/ganesha.conf
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 77;

# Exported path (mandatory)
Path = /mnt/dir1;

# Pseudo Path (required for NFS v4)
Pseudo = /mnt/dir1;

# Restrict the protocols that may use this export.  This cannot allow
# access that is denied in NFS_CORE_PARAM.
Protocols = 3,4;

# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;

# Whether to squash various users.
Squash = no_root_squash;

## Allowed security types for this export
#Sectype = sys,krb5,krb5i,krb5p;

# Exporting FSAL
FSAL {
    Name = VFS;
}

}

Started NFS-Ganesha using following command:
$ VMA_LOG_FILE=/var/log/ganesha/ganesha_libvma.log VMA_TRACELEVEL=DEBUG LD_PRELOAD=libvma.so /usr/bin/ganesha.nfsd -f /etc/ganesha/ganesha.conf -N NIV_EVENT

$vma_stats -p 208291
vmastat: stats for application: ganesha.nfsd with pid: 208291
fd ------------ total offloaded ------------- --------- total os ----------
pkt Kbyte eagain error poll% pkt Kbyte eagain error
poll Rx Ready:0/0 [os/offload] Timeouts:5761 Errors:0 Poll:0.00% Polling CPU:0%
epoll[61] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[64] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[67] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[81] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
poll Rx Ready:0/0 [os/offload] Timeouts:5761 Errors:0 Poll:0.00% Polling CPU:0%
epoll[61] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[64] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[67] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%
epoll[81] Rx Ready:0/0 [os/offload] Timeouts:19 Errors:0 Poll:0.00% Polling CPU:0%

In 'netstat' like view I do not see ganesha.nfsd as a 'Program name'
$ vma_stats -v 5
Proto Offloaded Recv-Q Send-Q Local Address Foreign Address State Inode PID/Program name
$

After looking at the log file I see following error/warning:
$ cat /var/log/ganesha/ganesha_libvma.log | more
Pid: 208291 Tid: 208291 VMA DEBUG: rdma_lib_reset doesn't exist returning 0
Pid: 208291 Tid: 208291 VMA DEBUG: EXIT: daemon() Child Process: starting with 208291
Pid: 208291 Tid: 208291 VMA DEBUG: sock_redirect_main()
Pid: 208291 Tid: 208291 VMA DEBUG: ENTER: close(fd=7)
Pid: 208291 Tid: 208291 VMA DEBUG: ENTER: close(fd=7)

After looking online for "rdma_lib_reset doesn't exist returning 0" I found a known issue:
1565428 Description: rdma_lib_reset function is not supported on the Upstream driver. Thus, fork() function is not supported. Does this mean that libvma will not work for the daemon process?

Then I restarted ganesha.nfsd with 'forground' option and the rdma_lib_reset() error in the log went away but still the IO traffic did not get offload. I see same Timeout messages in vma_stats output.

Any help would be much appreciated.

Thanks!