Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running vma inside docker container #1058

Open
Cha465 opened this issue Dec 21, 2023 · 9 comments
Open

Running vma inside docker container #1058

Cha465 opened this issue Dec 21, 2023 · 9 comments
Assignees
@AlexanderGrissik

Comments

@Cha465
Copy link

Cha465 commented Dec 21, 2023

Hi, is it viable to preload libvma inside docker container, in a loopback scenario (server and client run in the same container)? I preload libvma.so before executable, but the output of strace still shows sendmsg/recvmsg syscalls, and vma_stats finds no offloaded packets.
@igor-ivanov
Copy link
Collaborator

Hi @Cha465 the behavior is not container specific.
It is expected that scenario using tcp connection on the same server when both (server and client) processes under libvma whould work.

@Cha465
Copy link
Author

Cha465 commented Dec 25, 2023

Hi, I tried again with sockperf, specifying tcp connection:
LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf sr --tcp -i 0.0.0.0 // server
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf sr --tcp -i 0.0.0.0
VMA INFO: Current Time: Mon Dec 25 09:31:13 2023
VMA INFO: Pid: 77119
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf: [SERVER] listen on:
[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 77119] using recvfrom() to block on socket(s)

LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf pp -i 0.0.0.0 --tcp -t60 // client
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf pp -i 0.0.0.0 --tcp -t60
VMA INFO: Current Time: Mon Dec 25 09:31:31 2023
VMA INFO: Pid: 77125
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf : Warmup stage ( sending a few du mmy mess ERSION: 9.7.1-0 Developm ent Snap shot bui sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=60.000 sec; Warm up time=400 msec; SentMessages=4570324; ReceivedMessages=4570323
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=59.550 sec; SentMessages=4540388; ReceivedMessages=4540388
sockperf: ====> avg-latency=6.536 (std-dev=2.148)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 6.536 usec
sockperf: Total 4540388 observations; each percentile contains 45403.88 observations
sockperf: ---> observation = 2635.175
sockperf: ---> percentile 99.999 = 69.872
sockperf: ---> percentile 99.990 = 17.672
sockperf: ---> percentile 99.900 = 12.896
sockperf: ---> percentile 99.000 = 10.863
sockperf: ---> percentile 90.000 = 6.807
sockperf: ---> percentile 75.000 = 6.566
sockperf: ---> percentile 50.000 = 6.485
sockperf: ---> percentile 25.000 = 6.368
sockperf: ---> observation = 4.769

The performance is not ideal, and vma_stats -p <client_pid> is not showing a single line. What is the cause of this? Maybe the VMA ERROR must be fixed?

@igor-ivanov
Copy link
Collaborator

could you try to set specific ip instead of 0.0.0.0.

@Cha465
Copy link
Author

Cha465 commented Dec 26, 2023

127.0.0.1 or ip inside the docker container give the same result.

@igor-ivanov
Copy link
Collaborator

Of course, error should not be appeared.
Could you try on server first (w/o container) to separate potential issues.

Example on the same node:
Launch server

sudo env VMA_STATS_FILE=/tmp/libvma-sr.log LD_PRELOAD=libvma.so sockperf sr -i 1.1.1.10 --tcp

Statistic for server:

$ cat /tmp/libvma-sr.log
======================================================
        Fd=[24]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
- Foreign Address = [1.1.1.10:50613]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 1 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 2 / 313151 (100.00%) [miss/hit]
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
Rx poll: 3 / 0 (0.00%) [miss/hit]
======================================================

Launch client:

sudo env VMA_STATS_FILE=/tmp/libvma-cl.log LD_PRELOAD=libvma.so sockperf pp -i 1.1.1.10 --tcp

Statistic for client:

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:50613]
- Foreign Address = [1.1.1.10:11111]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 0 / 313152 (100.00%) [miss/hit]
======================================================

@Cha465
Copy link
Author

Cha465 commented Dec 26, 2023

I got these results:
server

cat /tmp/libvma-sr.log
======================================================
        Fd=[20]
- TCP, Blocked
Rx and Tx where not active
======================================================

client

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [0.0.0.0:48379]
Rx and Tx where not active
======================================================

@igor-ivanov
Copy link
Collaborator

Did you launch on Mellanox device? Did you see any error messages from VMA? Use direct ip as mentioned before.

@Cha465
Copy link
Author

Cha465 commented Dec 27, 2023

  • Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.
  • No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************
  • Yes, I used direct ip.

@igor-ivanov
Copy link
Collaborator

  • Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.

Just use ip that is set on Mellanox NIC

  • No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

It is strange. Did you use different users to set ulimit and launch application?

  • Yes, I used direct ip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlexanderGrissik AlexanderGrissik

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@igor-ivanov @Cha465 @AlexanderGrissik