Error When Running an air-land coupled case

[dongxthu]

Dear Team,

I submitted a land-atmosphere coupled case (compset: F2010-CICE, res: ne30pg2_EC30to60E2r2) using E3SM v3, it has been successfully built but failed to run. Note that another same case can run successfully and output data when using a single node, but it fails when using 12 nodes. Attached are the log files. I would greatly appreciate it if you could help me resolve this issue!

Best，
Dongze

atm.log.2326176.241119-173047.txt
cpl.log.2326176.241119-173047.txt
ice.log.2326176.241119-173047.txt
ocn.log.2326176.241119-173047.txt
e3sm.log.2326176.241119-173047.txt
lnd.log.2326176.241119-173047.txt

[rljacob]

The cpl.log shows it starts up and begins to integrate. But e3sm.log shows a network error:
[notch027:392880:0:392880] ib_mlx5_log.c:171 Local length on mlx5_0:1/IB (synd 0x1 vend 0x68 hw_synd 0/145)
[notch027:392880:0:392880] ib_mlx5_log.c:171 DCI QP 0x1c434 wqe[12641]: RDMA_READ s-- [rqpn 0x1fc23 rlid 55] [rva 0x1524316af610 rkey 0x1ec125] [va 0x146bd40e7010 len 4080 lkey 0x12da23]
==== backtrace (tid: 392880) ====
0 /uufs/chpc.utah.edu/sys/spack/v019/linux-rocky8-x86_64/gcc-11.2.0/ucx-1.15.0-p2hawz4uz7szc6f64lobyiyoyo6p7utg/lib/libucs.so.0(ucs_handle_error+0x294) [0x146bf20a01b4]

That's not from E3SM itself. Make sure you can run some simple parallel example code on 12 nodes.

[dongxthu]

Hi Robert, the case has been successfully run using 8 nodes. Thank you!!