Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gRPC stability issues #3313

Open
germa89 opened this issue Jul 26, 2024 · 10 comments
Open

gRPC stability issues #3313

germa89 opened this issue Jul 26, 2024 · 10 comments
Assignees
@germa89

Comments

@germa89
Copy link
Collaborator

germa89 commented Jul 26, 2024 •
edited
Loading

I'm going to use this issue to track and report progress.

Description

MAPDL tends to crash after several interations.

Prediscussion and assumptions

It has been suggested here that it might be related to a race conditio when writing the output file using (/out).

I do believe this issue #2386 is very related with all of this, so I'm testing something similar.

I should also check #1114

Testing framework

Running MAPDL on lyon5.

Connecting to it locally and run:

from ansys.mapdl.core import launch_mapdl

ip = "REDACTED"
port = 50052

mapdl = launch_mapdl(ip=ip, port=port)

from ansys.mapdl.core.examples import vmfiles

mapdl._ctrl("set_verb", 5)

with open(vmfiles['vm1'], "r") as fid:
    content = fid.read()

for i in range(8000):
    print(i)
    with mapdl.non_interactive:
        mapdl.input_strings(content)
        mapdl.clear()

Example results

crash file

Dumping the crash data for the current analysis.
    /ansys_inc/v242/ansys/lib/linx64/libansys.so(+0x3e5624c) [0x147e3725624c]
    /ansys_inc/v242/ansys/lib/linx64/libansys.so(+0xaa366b6) [0x147e3de366b6]
    /ansys_inc/v242/ansys/lib/linx64/libansys.so(sytrap11_+0x13) [0x147e3b3d9583]
    /ansys_inc/v242/tp/IntelCompiler/2023.1.0/linx64/lib/intel64/libifport.so.5(portlib_handler+0x2c) [0x147e5f7f13cc]
    /usr/lib64/libc.so.6(+0x3e6f0) [0x147e1f83e6f0]
    /usr/lib64/libc.so.6(free+0x1e) [0x147e1f899cfe]
    /usr/lib64/libc.so.6(+0x9bd37) [0x147e1f89bd37]
    /usr/lib64/libc.so.6(+0x89adb) [0x147e1f889adb]
    /usr/lib64/libc.so.6(+0x10ec40) [0x147e1f90ec40]

MAPDL stdout


===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 575617 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 575618 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 575619 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

Full output

Details 

 Beta activation of the GRPC server.

 Ansys Mechanical Enterprise                       


 *------------------------------------------------------------------*
 |                                                                  |
 |   W E L C O M E   T O   T H E   A N S Y S (R)  P R O G R A M     |
 |                                                                  |
 *------------------------------------------------------------------*




 ***************************************************************
 *         ANSYS MAPDL 2024 R2          LEGAL NOTICES          *
 ***************************************************************
 *                                                             *
 * Copyright 1971-2024 Ansys, Inc.  All rights reserved.       *
 * Unauthorized use, distribution or duplication is            *
 * prohibited.                                                 *
 *                                                             *
 * Ansys is a registered trademark of Ansys, Inc. or its       *
 * subsidiaries in the United States or other countries.       *
 * See the Ansys, Inc. online documentation or the Ansys, Inc. *
 * documentation CD or online help for the complete Legal      *
 * Notice.                                                     *
 *                                                             *
 ***************************************************************
 *                                                             *
 * THIS ANSYS SOFTWARE PRODUCT AND PROGRAM DOCUMENTATION       *
 * INCLUDE TRADE SECRETS AND CONFIDENTIAL AND PROPRIETARY      *
 * PRODUCTS OF ANSYS, INC., ITS SUBSIDIARIES, OR LICENSORS.    *
 * The software products and documentation are furnished by    *
 * Ansys, Inc. or its subsidiaries under a software license    *
 * agreement that contains provisions concerning               *
 * non-disclosure, copying, length and nature of use,          *
 * compliance with exporting laws, warranties, disclaimers,    *
 * limitations of liability, and remedies, and other           *
 * provisions.  The software products and documentation may be *
 * used, disclosed, transferred, or copied only in accordance  *
 * with the terms and conditions of that software license      *
 * agreement.                                                  *
 *                                                             *
 * Ansys, Inc. is a UL registered                              *
 * ISO 9001:2015 company.                                      *
 *                                                             *
 ***************************************************************
 *                                                             *
 * This product is subject to U.S. laws governing export and   *
 * re-export.                                                  *
 *                                                             *
 * For U.S. Government users, except as specifically granted   *
 * by the Ansys, Inc. software license agreement, the use,     *
 * duplication, or disclosure by the United States Government  *
 * is subject to restrictions stated in the Ansys, Inc.        *
 * software license agreement and FAR 12.212 (for non-DOD      *
 * licenses).                                                  *
 *                                                             *
 ***************************************************************



          *****  MAPDL COMMAND LINE ARGUMENTS  *****
  NONE

 RELEASE= 2024 R2              BUILD= 24.2BETA  UP20240603   VERSION=LINUX x64   
 CURRENT JOBNAME=file0  05:31:52  JUL 26, 2024 CP=      0.247


 RUN SETUP PROCEDURE FROM FILE= /ansys_inc/v242/ansys/apdl/start.ans

 >>> Using Custom Plugin Library : /home/gayuso/.git_repo/ansysDV/ansGRPC/build/lib/libGrpcMapdl.so

 Start GRPC Server
D0726 05:31:53.267729046  576183 config.cc:183]                        gRPC EXPERIMENT block_excessive_requests_before_settings_ack ON  (default:ON)
D0726 05:31:53.267897602  576183 config.cc:183]                        gRPC EXPERIMENT call_status_override_on_cancellation         OFF (default:OFF)
D0726 05:31:53.267902091  576183 config.cc:183]                        gRPC EXPERIMENT canary_client_privacy                        OFF (default:OFF)
D0726 05:31:53.267904571  576183 config.cc:183]                        gRPC EXPERIMENT chttp2_batch_requests                        ON  (default:ON)
D0726 05:31:53.267906561  576183 config.cc:183]                        gRPC EXPERIMENT chttp2_offload_on_rst_stream                 ON  (default:ON)
D0726 05:31:53.267908484  576183 config.cc:183]                        gRPC EXPERIMENT client_idleness                              ON  (default:ON)
D0726 05:31:53.267910198  576183 config.cc:183]                        gRPC EXPERIMENT client_privacy                               OFF (default:OFF)
D0726 05:31:53.267912269  576183 config.cc:183]                        gRPC EXPERIMENT combiner_offload_to_event_engine             ON  (default:ON)
D0726 05:31:53.267914190  576183 config.cc:183]                        gRPC EXPERIMENT event_engine_client                          OFF (default:OFF)
D0726 05:31:53.267915902  576183 config.cc:183]                        gRPC EXPERIMENT event_engine_dns                             OFF (default:OFF)
D0726 05:31:53.267917737  576183 config.cc:183]                        gRPC EXPERIMENT event_engine_listener                        ON  (default:ON)
D0726 05:31:53.267919571  576183 config.cc:183]                        gRPC EXPERIMENT free_large_allocator                         OFF (default:OFF)
D0726 05:31:53.267921315  576183 config.cc:183]                        gRPC EXPERIMENT http2_stats_fix                              ON  (default:ON)
D0726 05:31:53.267923157  576183 config.cc:183]                        gRPC EXPERIMENT keepalive_fix                                OFF (default:OFF)
D0726 05:31:53.267925126  576183 config.cc:183]                        gRPC EXPERIMENT keepalive_server_fix                         OFF (default:OFF)
D0726 05:31:53.267926840  576183 config.cc:183]                        gRPC EXPERIMENT lazier_stream_updates                        ON  (default:ON)
D0726 05:31:53.267928560  576183 config.cc:183]                        gRPC EXPERIMENT memory_pressure_controller                   OFF (default:OFF)
D0726 05:31:53.267930616  576183 config.cc:183]                        gRPC EXPERIMENT monitoring_experiment                        ON  (default:ON)
D0726 05:31:53.267932433  576183 config.cc:183]                        gRPC EXPERIMENT multiping                                    OFF (default:OFF)
D0726 05:31:53.267934121  576183 config.cc:183]                        gRPC EXPERIMENT overload_protection                          ON  (default:ON)
D0726 05:31:53.267935858  576183 config.cc:183]                        gRPC EXPERIMENT peer_state_based_framing                     OFF (default:OFF)
D0726 05:31:53.267937704  576183 config.cc:183]                        gRPC EXPERIMENT pending_queue_cap                            ON  (default:ON)
D0726 05:31:53.267939396  576183 config.cc:183]                        gRPC EXPERIMENT pick_first_happy_eyeballs                    ON  (default:ON)
D0726 05:31:53.267941202  576183 config.cc:183]                        gRPC EXPERIMENT ping_on_rst_stream                           ON  (default:ON)
D0726 05:31:53.267943210  576183 config.cc:183]                        gRPC EXPERIMENT promise_based_client_call                    OFF (default:OFF)
D0726 05:31:53.267944950  576183 config.cc:183]                        gRPC EXPERIMENT promise_based_inproc_transport               OFF (default:OFF)
D0726 05:31:53.267946924  576183 config.cc:183]                        gRPC EXPERIMENT promise_based_server_call                    OFF (default:OFF)
D0726 05:31:53.267948797  576183 config.cc:183]                        gRPC EXPERIMENT red_max_concurrent_streams                   OFF (default:OFF)
D0726 05:31:53.267950605  576183 config.cc:183]                        gRPC EXPERIMENT registered_method_lookup_in_transport        ON  (default:ON)
D0726 05:31:53.267952640  576183 config.cc:183]                        gRPC EXPERIMENT registered_methods_map                       OFF (default:OFF)
D0726 05:31:53.267954442  576183 config.cc:183]                        gRPC EXPERIMENT rfc_max_concurrent_streams                   OFF (default:OFF)
D0726 05:31:53.267956170  576183 config.cc:183]                        gRPC EXPERIMENT round_robin_delegate_to_pick_first           ON  (default:ON)
D0726 05:31:53.267958601  576183 config.cc:183]                        gRPC EXPERIMENT rstpit                                       OFF (default:OFF)
D0726 05:31:53.267960395  576183 config.cc:183]                        gRPC EXPERIMENT schedule_cancellation_over_write             OFF (default:OFF)
D0726 05:31:53.267962100  576183 config.cc:183]                        gRPC EXPERIMENT separate_ping_from_keepalive                 ON  (default:ON)
D0726 05:31:53.267964131  576183 config.cc:183]                        gRPC EXPERIMENT server_privacy                               OFF (default:OFF)
D0726 05:31:53.267965965  576183 config.cc:183]                        gRPC EXPERIMENT settings_timeout                             ON  (default:ON)
D0726 05:31:53.267967689  576183 config.cc:183]                        gRPC EXPERIMENT tarpit                                       ON  (default:ON)
D0726 05:31:53.267969671  576183 config.cc:183]                        gRPC EXPERIMENT tcp_frame_size_tuning                        OFF (default:OFF)
D0726 05:31:53.267971494  576183 config.cc:183]                        gRPC EXPERIMENT tcp_rcv_lowat                                OFF (default:OFF)
D0726 05:31:53.267973186  576183 config.cc:183]                        gRPC EXPERIMENT trace_record_callops                         OFF (default:OFF)
D0726 05:31:53.267975122  576183 config.cc:183]                        gRPC EXPERIMENT unconstrained_max_quota_buffer_size          OFF (default:OFF)
D0726 05:31:53.267976941  576183 config.cc:183]                        gRPC EXPERIMENT uniquely_unowned                             ON  (default:ON)
D0726 05:31:53.267978646  576183 config.cc:183]                        gRPC EXPERIMENT work_serializer_clears_time_cache            ON  (default:ON)
D0726 05:31:53.267980358  576183 config.cc:183]                        gRPC EXPERIMENT work_serializer_dispatch                     OFF (default:OFF)
D0726 05:31:53.267982165  576183 config.cc:183]                        gRPC EXPERIMENT write_size_cap                               ON  (default:ON)
D0726 05:31:53.267983842  576183 config.cc:183]                        gRPC EXPERIMENT write_size_policy                            ON  (default:ON)
D0726 05:31:53.267985525  576183 config.cc:183]                        gRPC EXPERIMENT wrr_delegate_to_pick_first                   ON  (default:ON)
I0726 05:31:53.268462018  576183 ev_epoll1_linux.cc:123]               grpc epoll fd: 34
D0726 05:31:53.268487240  576183 ev_posix.cc:113]                      Using polling engine: epoll1
I0726 05:31:53.268705523  576183 init.cc:117]                          grpc_init(void)
I0726 05:31:53.268785907  576183 init.cc:117]                          grpc_init(void)
I0726 05:31:53.268811214  576183 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=0, polling_type=0)
I0726 05:31:53.268837528  576183 server_builder.cc:352]                Synchronous server. Num CQs: 1, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10000
I0726 05:31:53.268846692  576183 init.cc:117]                          grpc_init(void)
I0726 05:31:53.269011273  576183 server.cc:1901]                       grpc_server_create(0x7ffdba6e08d0, (nil))
D0726 05:31:53.269108046  576183 lb_policy_registry.cc:46]             registering LB policy factory for "priority_experimental"
D0726 05:31:53.269128735  576183 lb_policy_registry.cc:46]             registering LB policy factory for "outlier_detection_experimental"
D0726 05:31:53.269140081  576183 lb_policy_registry.cc:46]             registering LB policy factory for "weighted_target_experimental"
D0726 05:31:53.269148470  576183 lb_policy_registry.cc:46]             registering LB policy factory for "pick_first"
D0726 05:31:53.269173241  576183 lb_policy_registry.cc:46]             registering LB policy factory for "round_robin"
D0726 05:31:53.269183062  576183 lb_policy_registry.cc:46]             registering LB policy factory for "weighted_round_robin"
D0726 05:31:53.269312115  576183 lb_policy_registry.cc:46]             registering LB policy factory for "grpclb"
D0726 05:31:53.269390870  576183 dns_resolver_plugin.cc:50]            Using ares dns resolver
D0726 05:31:53.269455757  576183 lb_policy_registry.cc:46]             registering LB policy factory for "rls_experimental"
D0726 05:31:53.269564465  576183 lb_policy_registry.cc:46]             registering LB policy factory for "xds_cluster_manager_experimental"
D0726 05:31:53.269574119  576183 lb_policy_registry.cc:46]             registering LB policy factory for "xds_cluster_impl_experimental"
D0726 05:31:53.269581946  576183 lb_policy_registry.cc:46]             registering LB policy factory for "cds_experimental"
D0726 05:31:53.269589753  576183 lb_policy_registry.cc:46]             registering LB policy factory for "xds_cluster_resolver_experimental"
D0726 05:31:53.269602851  576183 lb_policy_registry.cc:46]             registering LB policy factory for "xds_override_host_experimental"
D0726 05:31:53.269613233  576183 lb_policy_registry.cc:46]             registering LB policy factory for "xds_wrr_locality_experimental"
D0726 05:31:53.269620802  576183 lb_policy_registry.cc:46]             registering LB policy factory for "ring_hash_experimental"
D0726 05:31:53.269638291  576183 certificate_provider_registry.cc:33]  registering certificate provider factory for "file_watcher"
I0726 05:31:53.269952662  576183 init.cc:117]                          grpc_init(void)
I0726 05:31:53.271474718  576183 ev_epoll1_linux.cc:359]               grpc epoll fd: 36
I0726 05:31:53.271712984  576183 server.cc:2017]                       grpc_server_set_config_fetcher(server=0x193a640, config_fetcher=(nil))
I0726 05:31:53.271734791  576183 server.cc:1914]                       grpc_server_register_completion_queue(server=0x193a640, cq=0x18e2a70, reserved=(nil))
I0726 05:31:53.271763550  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/SendCommand, host=(null), flags=0x00000000)
I0726 05:31:53.271783697  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/SendCommandS, host=(null), flags=0x00000000)
I0726 05:31:53.271787085  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/InputFileLocal, host=(null), flags=0x00000000)
I0726 05:31:53.271789964  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/InputFile, host=(null), flags=0x00000000)
I0726 05:31:53.271792622  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/InputFileS, host=(null), flags=0x00000000)
I0726 05:31:53.271794958  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/FctCall, host=(null), flags=0x00000000)
I0726 05:31:53.271797378  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetNodes, host=(null), flags=0x00000000)
I0726 05:31:53.271800061  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetNode, host=(null), flags=0x00000000)
I0726 05:31:53.271802450  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/SetNode, host=(null), flags=0x00000000)
I0726 05:31:53.271805177  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/Nodes, host=(null), flags=0x00000000)
I0726 05:31:53.271807692  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetElements, host=(null), flags=0x00000000)
I0726 05:31:53.271809853  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetElement, host=(null), flags=0x00000000)
I0726 05:31:53.271812224  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/LoadElements, host=(null), flags=0x00000000)
I0726 05:31:53.271814643  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetElementType, host=(null), flags=0x00000000)
I0726 05:31:53.271817209  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/LoadElementTypeDescription, host=(null), flags=0x00000000)
I0726 05:31:53.271819529  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetParameter, host=(null), flags=0x00000000)
I0726 05:31:53.271821939  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/GetVariable, host=(null), flags=0x00000000)
I0726 05:31:53.271824279  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/Get, host=(null), flags=0x00000000)
I0726 05:31:53.271826833  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/VGet, host=(null), flags=0x00000000)
I0726 05:31:53.271829616  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/VGet2, host=(null), flags=0x00000000)
I0726 05:31:53.271832706  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, method=/ansys.api.mapdl.v0.MapdlService/Ctrl, host=(null), flags=0x00000000)
I0726 05:31:53.271835228  576183 server.cc:1935]                       grpc_server_register_method(server=0x193a640, 
...

REDUCED

...

 >> Received Command : __PYMAPDL_SESSION_ID__='f3154265a79e4d4a834fb5b9476'

 /OUTPUT FILE=         
 >> Msg Size: 72
I0726 05:32:15.453184460  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3b00055b0, ops=0x14e3e43d9220, nops=3, tag=0x14e3e43d95e0, reserved=(nil))
I0726 05:32:15.453192129  576313 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 05:32:15.453194828  576313 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x14e3b0004ef0
I0726 05:32:15.453197424  576313 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 05:32:15.453235603  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3b00017c0, tag=0x14e3b00081c0, error=OK, done=0x14e401b69890, done_arg=0x14e3b0008320, storage=0x14e3b0008370)
I0726 05:32:15.453241218  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3b00017c0, tag=0x14e3e43d95e0, error=OK, done=0x14e401b69890, done_arg=0x14e3b00083f0, storage=0x14e3b0008440)
I0726 05:32:15.453245640  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3b00017c0]: OP_COMPLETE: tag:0x14e3e43d95e0 OK
I0726 05:32:15.453249666  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3b00017c0)
I0726 05:32:15.453253322  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3b00017c0]: OP_COMPLETE: tag:0x14e3b00081c0 OK
I0726 05:32:15.453256227  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3b00017c0]: QUEUE_SHUTDOWN
I0726 05:32:15.453258984  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3b00014b8)
I0726 05:32:15.453261401  576313 call.cc:967]                          grpc_call_unref(c=0x14e3b00055b0)
I0726 05:32:15.453264221  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3b0006588)
I0726 05:32:15.453272842  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3b00012a0)
I0726 05:32:15.453275816  576313 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x14e3b00017c0)
I0726 05:32:15.453277736  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3b00017c0)
I0726 05:32:15.453281388  576313 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x18e2a70, deadline=gpr_timespec { tv_sec: 611704, tv_nsec: 752449669, clock_type: 0 }, reserved=(nil))
I0726 05:32:15.580727783  576237 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x14e3cc002048
I0726 05:32:15.580787218  576237 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 05:32:15.580793159  576237 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x14e3cc0048d0)
I0726 05:32:15.580799114  576237 completion_queue.cc:710]              cq_end_op_for_next(cq=0x18e2a70, tag=0x14e3cc004890, error=OK, done=0x14e401b935b0, done_arg=0x14e3cc004ff0, storage=0x14e3cc005018)
I0726 05:32:15.580872562  576313 completion_queue.cc:1069]             RETURN_EVENT[0x18e2a70]: OP_COMPLETE: tag:0x14e3cc004890 OK
I0726 05:32:15.580885379  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3cc001070, ops=0x14e3e43d9810, nops=1, tag=0x14e3cc003b80, reserved=(nil))
I0726 05:32:15.580890245  576313 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x14e3cc003bec
I0726 05:32:15.580916680  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3cc001070, ops=0x14e3e43d8bc0, nops=1, tag=0x14e3e43d91d8, reserved=(nil))
I0726 05:32:15.580920559  576313 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x14e3e43d91f0
I0726 05:32:15.601657369  576239 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3cc004df0, tag=0x14e3e43d91d8, error=OK, done=0x14e401b69890, done_arg=0x14e3cc003db0, storage=0x14e3cc003e00)
I0726 05:32:15.601746707  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3cc004df0]: OP_COMPLETE: tag:0x14e3e43d91d8 OK
 >> Open File tmp_nsvpbqdfiv.inp
I0726 05:32:15.604796292  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3cc001070, ops=0x14e3e43d8bc0, nops=1, tag=0x14e3e43d9358, reserved=(nil))
I0726 05:32:15.604816452  576313 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x14e3e43d9370
I0726 05:32:15.604847049  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3cc004df0, tag=0x14e3e43d9358, error=OK, done=0x14e401b69890, done_arg=0x14e3cc003db0, storage=0x14e3cc003e00)
I0726 05:32:15.604861595  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3cc004df0]: OP_COMPLETE: tag:0x14e3e43d9358 OK
 >> File generated
I0726 05:32:15.606636677  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3cc001070, ops=0x14e3e43d9260, nops=3, tag=0x14e3e43d95e0, reserved=(nil))
I0726 05:32:15.606641650  576313 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 05:32:15.606644029  576313 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x14e3c4004b10
I0726 05:32:15.606646615  576313 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 05:32:15.606679141  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3cc004df0, tag=0x14e3cc003b80, error=OK, done=0x14e401b69890, done_arg=0x14e3cc003ce0, storage=0x14e3cc003d30)
I0726 05:32:15.606684745  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3cc004df0, tag=0x14e3e43d95e0, error=OK, done=0x14e401b69890, done_arg=0x14e3cc003e80, storage=0x14e3cc003ed0)
I0726 05:32:15.606688556  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3cc004df0]: OP_COMPLETE: tag:0x14e3e43d95e0 OK
I0726 05:32:15.606692611  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3cc004df0)
I0726 05:32:15.606696114  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3cc004df0]: OP_COMPLETE: tag:0x14e3cc003b80 OK
I0726 05:32:15.606699037  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3cc004df0]: QUEUE_SHUTDOWN
I0726 05:32:15.606701625  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3cc004ae8)
I0726 05:32:15.606704050  576313 call.cc:967]                          grpc_call_unref(c=0x14e3cc001070)
I0726 05:32:15.606706629  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3cc002048)
I0726 05:32:15.606715035  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3cc0048d0)
I0726 05:32:15.606718058  576313 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x14e3cc004df0)
I0726 05:32:15.606719923  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3cc004df0)
I0726 05:32:15.606723970  576313 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x18e2a70, deadline=gpr_timespec { tv_sec: 611704, tv_nsec: 905892018, clock_type: 0 }, reserved=(nil))
I0726 05:32:15.742868233  576235 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x14e3d4001fc8
I0726 05:32:15.742898932  576235 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 05:32:15.742904393  576235 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x14e3d4005610)
I0726 05:32:15.742910081  576235 completion_queue.cc:710]              cq_end_op_for_next(cq=0x18e2a70, tag=0x14e3d40055d0, error=OK, done=0x14e401b935b0, done_arg=0x14e3d4005150, storage=0x14e3d4005178)
I0726 05:32:15.742990581  576313 completion_queue.cc:1069]             RETURN_EVENT[0x18e2a70]: OP_COMPLETE: tag:0x14e3d40055d0 OK
I0726 05:32:15.743002983  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3d4000ff0, ops=0x14e3e43d9810, nops=1, tag=0x14e3d4003b00, reserved=(nil))
I0726 05:32:15.743007706  576313 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x14e3d4003b6c
I0726 05:32:15.743030871  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3d4000ff0, ops=0x14e3e43d8bc0, nops=1, tag=0x14e3e43d91d8, reserved=(nil))
I0726 05:32:15.743034375  576313 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x14e3e43d91f0
I0726 05:32:15.744710066  576236 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3d40045d0, tag=0x14e3e43d91d8, error=OK, done=0x14e401b69890, done_arg=0x14e3d4003d30, storage=0x14e3d4003d80)
 >> Open File _input_tmp_kyvjqebjxt_.inp
I0726 05:32:15.744793817  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3d40045d0]: OP_COMPLETE: tag:0x14e3e43d91d8 OK
I0726 05:32:15.747683958  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3d4000ff0, ops=0x14e3e43d8bc0, nops=1, tag=0x14e3e43d9358, reserved=(nil))
I0726 05:32:15.747704043  576313 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x14e3e43d9370
I0726 05:32:15.747732814  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3d40045d0, tag=0x14e3e43d9358, error=OK, done=0x14e401b69890, done_arg=0x14e3d4003d30, storage=0x14e3d4003d80)
I0726 05:32:15.747746527  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3d40045d0]: OP_COMPLETE: tag:0x14e3e43d9358 OK
 >> File generated
I0726 05:32:15.749682608  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3d4000ff0, ops=0x14e3e43d9260, nops=3, tag=0x14e3e43d95e0, reserved=(nil))
I0726 05:32:15.749688022  576313 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 05:32:15.749690332  576313 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x14e3c8004560
I0726 05:32:15.749692840  576313 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 05:32:15.749724157  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3d40045d0, tag=0x14e3d4003b00, error=OK, done=0x14e401b69890, done_arg=0x14e3d4003c60, storage=0x14e3d4003cb0)
I0726 05:32:15.749729830  576313 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3d40045d0, tag=0x14e3e43d95e0, error=OK, done=0x14e401b69890, done_arg=0x14e3d4003e00, storage=0x14e3d4003e50)
I0726 05:32:15.749734276  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3d40045d0]: OP_COMPLETE: tag:0x14e3e43d95e0 OK
I0726 05:32:15.749737889  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3d40045d0)
I0726 05:32:15.749741030  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3d40045d0]: OP_COMPLETE: tag:0x14e3d4003b00 OK
I0726 05:32:15.749743830  576313 completion_queue.cc:1299]             RETURN_EVENT[0x14e3d40045d0]: QUEUE_SHUTDOWN
I0726 05:32:15.749746324  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3d4005828)
I0726 05:32:15.749748853  576313 call.cc:967]                          grpc_call_unref(c=0x14e3d4000ff0)
I0726 05:32:15.749751355  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3d4001fc8)
I0726 05:32:15.749760081  576313 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x14e3d4005610)
I0726 05:32:15.749763517  576313 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x14e3d40045d0)
I0726 05:32:15.749765641  576313 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x14e3d40045d0)
I0726 05:32:15.749769633  576313 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x18e2a70, deadline=gpr_timespec { tv_sec: 611705, tv_nsec: 48937736, clock_type: 0 }, reserved=(nil))
I0726 05:32:15.893281281  576233 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x14e3dc006308
I0726 05:32:15.893321756  576233 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x14e3dc0062e0
I0726 05:32:15.893335608  576233 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 05:32:15.893340392  576233 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x14e3dc001310)
I0726 05:32:15.893345557  576233 completion_queue.cc:710]              cq_end_op_for_next(cq=0x18e2a70, tag=0x14e3dc0012d0, error=OK, done=0x14e401b935b0, done_arg=0x14e3dc004e00, storage=0x14e3dc004e28)
I0726 05:32:15.893421706  576313 completion_queue.cc:1069]             RETURN_EVENT[0x18e2a70]: OP_COMPLETE: tag:0x14e3dc0012d0 OK
I0726 05:32:15.893474673  576313 call.cc:3710]                         grpc_call_start_batch(call=0x14e3dc005330, ops=0x14e3e43d9810, nops=1, tag=0x14e3dc0081d0, reserved=(nil))
I0726 05:32:15.893479577  576313 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x14e3dc00823c

 /OUTPUT FILE=         
I0726 05:32:15.949337504  576348 call.cc:3710]                         grpc_call_start_batch(call=0x14e3dc005330, ops=0x14e364ff1460, nops=2, tag=0x14e3dc0015e0, reserved=(nil))
I0726 05:32:15.949353557  576348 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 05:32:15.949357466  576348 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x14e34c000ef0
I0726 05:32:15.949402147  576348 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x14e3dc000f90, tag=0x14e3dc0015e0, error=OK, done=0x14e401b69890, done_arg=0x14e3dc008400, storage=0x14e3dc008450)
I0726 05:32:15.949411289  576348 completion_queue.cc:1299]             RETURN_EVENT[0x14e3dc000f90]: OP_COMPLETE: tag:0x14e3dc0015e0 OK
/ansys_inc/v242/ansys/bin/ansysdis242: line 77: 576183 Segmentation fault      (core dumped) /ansys_inc/v242/ansys/bin/linx64/ansys.e -grpc

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 575617 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 575618 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 575619 RUNNING AT lyolinlst5.ansys.com
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

@germa89 germa89 mentioned this issue Jul 26, 2024
Open
18 tasks
@germa89 germa89 self-assigned this Jul 26, 2024
@germa89
Copy link
Collaborator Author

germa89 commented Jul 26, 2024

I'm tracking down the issue to: Mapdl::InputFileS

@germa89
Copy link
Collaborator Author

germa89 commented Jul 26, 2024

In another run, the fail happened either while waiting for the thread (join) or when deleting it.

It might be the second one.

More info:

I0726 07:12:46.125645533  592846 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x150e00007e6c
 >> Received Command : /DELETE,tmp_ezkhtqopya.inp,,,

 /OUTPUT FILE=         
 >> Msg Size: 40
I0726 07:12:46.155433021  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150e000051f0, ops=0x150e10bdd220, nops=3, tag=0x150e10bdd5e0, reserved=(nil))
I0726 07:12:46.155443836  592846 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 07:12:46.155447260  592846 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x150e00004870
I0726 07:12:46.155450084  592846 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 07:12:46.155508894  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e000049c0, tag=0x150e00007e00, error=OK, done=0x150e3e379890, done_arg=0x150e00007f60, storage=0x150e00007fb0)
I0726 07:12:46.155516675  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e000049c0, tag=0x150e10bdd5e0, error=OK, done=0x150e3e379890, done_arg=0x150e00008030, storage=0x150e00008080)
I0726 07:12:46.155522426  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e000049c0]: OP_COMPLETE: tag:0x150e10bdd5e0 OK
I0726 07:12:46.155527646  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150e000049c0)
I0726 07:12:46.155531608  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e000049c0]: OP_COMPLETE: tag:0x150e00007e00 OK
I0726 07:12:46.155534778  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e000049c0]: QUEUE_SHUTDOWN
I0726 07:12:46.155537564  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e00001168)
I0726 07:12:46.155540117  592846 call.cc:967]                          grpc_call_unref(c=0x150e000051f0)
I0726 07:12:46.155543101  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e000061c8)
I0726 07:12:46.155555437  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e00000f50)
I0726 07:12:46.155559362  592846 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x150e000049c0)
I0726 07:12:46.155561297  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150e000049c0)
I0726 07:12:46.155565419  592846 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x185bcb0, deadline=gpr_timespec { tv_sec: 617735, tv_nsec: 454733457, clock_type: 0 }, reserved=(nil))
I0726 07:12:46.283204648  592766 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x150de40062c8
I0726 07:12:46.283269132  592766 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150de40062a0
I0726 07:12:46.283301731  592766 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 07:12:46.283307051  592766 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x150de4004900)
I0726 07:12:46.283312271  592766 completion_queue.cc:710]              cq_end_op_for_next(cq=0x185bcb0, tag=0x150de40048c0, error=OK, done=0x150e3e3a35b0, done_arg=0x150de40010b0, storage=0x150de40010d8)
I0726 07:12:46.283392630  592846 completion_queue.cc:1069]             RETURN_EVENT[0x185bcb0]: OP_COMPLETE: tag:0x150de40048c0 OK
I0726 07:12:46.283410667  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150de40052f0, ops=0x150e10bdd810, nops=1, tag=0x150de4007f00, reserved=(nil))
I0726 07:12:46.283415966  592846 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x150de4007f6c
 >> Received Command : __PYMAPDL_SESSION_ID__='c9c33bd67d464f33b04b8d7a947'

 /OUTPUT FILE=         
 >> Msg Size: 72
I0726 07:12:46.290713181  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150de40052f0, ops=0x150e10bdd220, nops=3, tag=0x150e10bdd5e0, reserved=(nil))
I0726 07:12:46.290731054  592846 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 07:12:46.290738425  592846 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x150e00004870
I0726 07:12:46.290745489  592846 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 07:12:46.290833731  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150de4004e20, tag=0x150de4007f00, error=OK, done=0x150e3e379890, done_arg=0x150de4008060, storage=0x150de40080b0)
I0726 07:12:46.290850374  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150de4004e20, tag=0x150e10bdd5e0, error=OK, done=0x150e3e379890, done_arg=0x150de4008130, storage=0x150de4008180)
I0726 07:12:46.290862377  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150de4004e20]: OP_COMPLETE: tag:0x150e10bdd5e0 OK
I0726 07:12:46.290872722  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150de4004e20)
I0726 07:12:46.290882226  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150de4004e20]: OP_COMPLETE: tag:0x150de4007f00 OK
I0726 07:12:46.290889530  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150de4004e20]: QUEUE_SHUTDOWN
I0726 07:12:46.290897105  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150de4004b18)
I0726 07:12:46.290904051  592846 call.cc:967]                          grpc_call_unref(c=0x150de40052f0)
I0726 07:12:46.290911660  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150de40062c8)
I0726 07:12:46.290931772  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150de4004900)
I0726 07:12:46.290939304  592846 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x150de4004e20)
I0726 07:12:46.290945142  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150de4004e20)
I0726 07:12:46.290956556  592846 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x185bcb0, deadline=gpr_timespec { tv_sec: 617735, tv_nsec: 590122631, clock_type: 0 }, reserved=(nil))
I0726 07:12:46.435088177  592762 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x150df40062c8
I0726 07:12:46.435149850  592762 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 07:12:46.435178654  592762 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x150df4004530)
I0726 07:12:46.435192186  592762 completion_queue.cc:710]              cq_end_op_for_next(cq=0x185bcb0, tag=0x150df40044f0, error=OK, done=0x150e3e3a35b0, done_arg=0x150df4005060, storage=0x150df4005088)
I0726 07:12:46.435271734  592846 completion_queue.cc:1069]             RETURN_EVENT[0x185bcb0]: OP_COMPLETE: tag:0x150df40044f0 OK
I0726 07:12:46.435300243  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150df40052f0, ops=0x150e10bdd810, nops=1, tag=0x150df4007e00, reserved=(nil))
I0726 07:12:46.435311470  592846 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x150df4007e6c
I0726 07:12:46.435366099  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150df40052f0, ops=0x150e10bdcbc0, nops=1, tag=0x150e10bdd1d8, reserved=(nil))
I0726 07:12:46.435375559  592846 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150e10bdd1f0
I0726 07:12:46.515994160  592782 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 515982833, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516009628  592780 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516001378, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516004315  592776 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 515992831, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516018757  592774 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516010607, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516087958  592783 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516079969, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516090318  592772 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516082325, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516106139  592779 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516098100, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516115230  592781 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516111659, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516136035  592778 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516128189, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516192041  592777 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516183708, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.516964371  592775 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 516959559, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.517957758  592773 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992367, tv_nsec: 517955391, clock_type: 1 }, reserved=(nil))
I0726 07:12:46.573224336  592768 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150df4004a50, tag=0x150e10bdd1d8, error=OK, done=0x150e3e379890, done_arg=0x150df4008030, storage=0x150df4008080)
I0726 07:12:46.573325602  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150df4004a50]: OP_COMPLETE: tag:0x150e10bdd1d8 OK
 >> Open File tmp_yhmzjiqnwb.inp
I0726 07:12:46.574772302  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150df40052f0, ops=0x150e10bdcbc0, nops=1, tag=0x150e10bdd358, reserved=(nil))
I0726 07:12:46.574780467  592846 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150e10bdd370
I0726 07:12:46.585773173  592755 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150df4004a50, tag=0x150e10bdd358, error=OK, done=0x150e3e379890, done_arg=0x150df4008030, storage=0x150df4008080)
I0726 07:12:46.585818947  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150df4004a50]: OP_COMPLETE: tag:0x150e10bdd358 OK
 >> File generated
I0726 07:12:46.586413845  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150df40052f0, ops=0x150e10bdd260, nops=3, tag=0x150e10bdd5e0, reserved=(nil))
I0726 07:12:46.586418917  592846 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 07:12:46.586421280  592846 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x150e00004870
I0726 07:12:46.586423856  592846 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 07:12:46.586457264  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150df4004a50, tag=0x150df4007e00, error=OK, done=0x150e3e379890, done_arg=0x150df4007f60, storage=0x150df4007fb0)
I0726 07:12:46.586462913  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150df4004a50, tag=0x150e10bdd5e0, error=OK, done=0x150e3e379890, done_arg=0x150df4008100, storage=0x150df4008150)
I0726 07:12:46.586466720  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150df4004a50]: OP_COMPLETE: tag:0x150e10bdd5e0 OK
I0726 07:12:46.586470425  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150df4004a50)
I0726 07:12:46.586473568  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150df4004a50]: OP_COMPLETE: tag:0x150df4007e00 OK
I0726 07:12:46.586476367  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150df4004a50]: QUEUE_SHUTDOWN
I0726 07:12:46.586478807  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150df4004748)
I0726 07:12:46.586481215  592846 call.cc:967]                          grpc_call_unref(c=0x150df40052f0)
I0726 07:12:46.586483869  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150df40062c8)
I0726 07:12:46.586492013  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150df4004530)
I0726 07:12:46.586494893  592846 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x150df4004a50)
I0726 07:12:46.586497040  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150df4004a50)
I0726 07:12:46.586501087  592846 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x185bcb0, deadline=gpr_timespec { tv_sec: 617735, tv_nsec: 885669370, clock_type: 0 }, reserved=(nil))
I0726 07:12:47.355654865  592757 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x150e0c006988
I0726 07:12:47.355717554  592757 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 07:12:47.355726926  592757 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x150e0c0052f0)
I0726 07:12:47.355734977  592757 completion_queue.cc:710]              cq_end_op_for_next(cq=0x185bcb0, tag=0x150e0c0052b0, error=OK, done=0x150e3e3a35b0, done_arg=0x150e0c004c00, storage=0x150e0c004c28)
I0726 07:12:47.355805663  592846 completion_queue.cc:1069]             RETURN_EVENT[0x185bcb0]: OP_COMPLETE: tag:0x150e0c0052b0 OK
I0726 07:12:47.355825236  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150e0c0059b0, ops=0x150e10bdd810, nops=1, tag=0x150e0c0084c0, reserved=(nil))
I0726 07:12:47.355831023  592846 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x150e0c00852c
I0726 07:12:47.355886267  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150e0c0059b0, ops=0x150e10bdcbc0, nops=1, tag=0x150e10bdd1d8, reserved=(nil))
I0726 07:12:47.355890988  592846 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150e10bdd1f0
I0726 07:12:47.359560590  592764 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e0c004a00, tag=0x150e10bdd1d8, error=OK, done=0x150e3e379890, done_arg=0x150e0c0086f0, storage=0x150e0c008740)
I0726 07:12:47.359651878  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e0c004a00]: OP_COMPLETE: tag:0x150e10bdd1d8 OK
 >> Open File _input_tmp_fwqhsgxfkw_.inp
I0726 07:12:47.361279615  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150e0c0059b0, ops=0x150e10bdcbc0, nops=1, tag=0x150e10bdd358, reserved=(nil))
I0726 07:12:47.361288144  592846 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150e10bdd370
I0726 07:12:47.361301455  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e0c004a00, tag=0x150e10bdd358, error=OK, done=0x150e3e379890, done_arg=0x150e0c0086f0, storage=0x150e0c008740)
I0726 07:12:47.361307719  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e0c004a00]: OP_COMPLETE: tag:0x150e10bdd358 OK
 >> File generated
I0726 07:12:47.363793787  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150e0c0059b0, ops=0x150e10bdd260, nops=3, tag=0x150e10bdd5e0, reserved=(nil))
I0726 07:12:47.363799342  592846 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 07:12:47.363801663  592846 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x150e00004870
I0726 07:12:47.363804277  592846 call.cc:1553]                         ops[2]: SEND_STATUS_FROM_SERVER status=0 details=(null)(nil)
I0726 07:12:47.363854303  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e0c004a00, tag=0x150e0c0084c0, error=OK, done=0x150e3e379890, done_arg=0x150e0c008620, storage=0x150e0c008670)
I0726 07:12:47.363861167  592846 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150e0c004a00, tag=0x150e10bdd5e0, error=OK, done=0x150e3e379890, done_arg=0x150e0c0087c0, storage=0x150e0c008810)
I0726 07:12:47.363864971  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e0c004a00]: OP_COMPLETE: tag:0x150e10bdd5e0 OK
I0726 07:12:47.363869113  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150e0c004a00)
I0726 07:12:47.363872744  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e0c004a00]: OP_COMPLETE: tag:0x150e0c0084c0 OK
I0726 07:12:47.363875421  592846 completion_queue.cc:1299]             RETURN_EVENT[0x150e0c004a00]: QUEUE_SHUTDOWN
I0726 07:12:47.363878145  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e0c005508)
I0726 07:12:47.363880656  592846 call.cc:967]                          grpc_call_unref(c=0x150e0c0059b0)
I0726 07:12:47.363883147  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e0c006988)
I0726 07:12:47.363898215  592846 metadata_array.cc:35]                 grpc_metadata_array_destroy(array=0x150e0c0052f0)
I0726 07:12:47.363901768  592846 completion_queue.cc:1403]             grpc_completion_queue_destroy(cq=0x150e0c004a00)
I0726 07:12:47.363903664  592846 completion_queue.cc:1398]             grpc_completion_queue_shutdown(cq=0x150e0c004a00)
I0726 07:12:47.363908302  592846 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x185bcb0, deadline=gpr_timespec { tv_sec: 617736, tv_nsec: 663076263, clock_type: 0 }, reserved=(nil))
I0726 07:12:47.515266432  592765 call.cc:1553]                         ops[0]: RECV_INITIAL_METADATA ptr=0x150df0006588
I0726 07:12:47.515333109  592765 call.cc:1553]                         ops[0]: RECV_MESSAGE ptr=0x150df0006560
I0726 07:12:47.515352885  592765 completion_queue.cc:521]              grpc_completion_queue_create_internal(completion_type=1, polling_type=0)
I0726 07:12:47.515360519  592765 metadata_array.cc:30]                 grpc_metadata_array_init(array=0x150df0001790)
I0726 07:12:47.515367937  592765 completion_queue.cc:710]              cq_end_op_for_next(cq=0x185bcb0, tag=0x150df0001750, error=OK, done=0x150e3e3a35b0, done_arg=0x150df0001eb0, storage=0x150df0001ed8)
I0726 07:12:47.515385958  592846 completion_queue.cc:1069]             RETURN_EVENT[0x185bcb0]: OP_COMPLETE: tag:0x150df0001750 OK
I0726 07:12:47.515407163  592846 call.cc:3710]                         grpc_call_start_batch(call=0x150df00055b0, ops=0x150e10bdd810, nops=1, tag=0x150df0008450, reserved=(nil))
I0726 07:12:47.515412219  592846 call.cc:1553]                         ops[0]: RECV_CLOSE_ON_SERVER cancelled=0x150df00084bc
 >> Entering: Mapdl::InputFileS
 >> InputFile Name : _input_tmp_fwqhsgxfkw_.inp
 >> Mapdl::InputFileS: Not mute.

 /OUTPUT FILE=         
I0726 07:12:47.570766831  592984 call.cc:3710]                         grpc_call_start_batch(call=0x150df00055b0, ops=0x150e109dc460, nops=2, tag=0x150df0001a60, reserved=(nil))
I0726 07:12:47.570783782  592984 call.cc:1553]                         ops[0]: SEND_INITIAL_METADATA(nil)
I0726 07:12:47.570787278  592984 call.cc:1553]                         ops[1]: SEND_MESSAGE ptr=0x150d88001180
I0726 07:12:47.570834558  592984 completion_queue.cc:784]              cq_end_op_for_pluck(cq=0x150df0001cb0, tag=0x150df0001a60, error=OK, done=0x150e3e379890, done_arg=0x150df0008680, storage=0x150df00086d0)
I0726 07:12:47.570844429  592984 completion_queue.cc:1299]             RETURN_EVENT[0x150df0001cb0]: OP_COMPLETE: tag:0x150df0001a60 OK
I0726 07:12:47.616965839  592776 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 616959995, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.616972546  592777 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 616967004, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.616979852  592780 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 616975833, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.616987025  592774 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 616982749, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.616994259  592782 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 616990178, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617020602  592783 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617016532, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617032718  592781 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617028567, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617060243  592779 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617056524, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617065905  592778 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617063902, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617083323  592772 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617079767, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.617950739  592775 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 617948157, clock_type: 1 }, reserved=(nil))
I0726 07:12:47.618953630  592773 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992368, tv_nsec: 618949750, clock_type: 1 }, reserved=(nil))
 >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
 >> Mapdl::InputFileS: Running command: '/POST1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         '
   *****MAPDL VERIFICATION RUN ONLY*****
     DO NOT USE RESULTS FOR PRODUCTION

          ***** MAPDL RESULTS INTERPRETATION (POST1) *****


 USE LAST SUBSTEP ON RESULT FILE  FOR LOAD CASE 0 

 SET COMMAND GOT LOAD STEP=     1  SUBSTEP=     1  CUMULATIVE ITERATION=     1
   TIME/FREQUENCY=  1.0000    
 TITLE=  VM1, STATICALLY INDETERMINATE REACTION FORCE ANALYSIS                  
 >> Mapdl::InputFileS: Running command: 'NSEL,S,LOC,Y,10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                '

 SELECT       FOR ITEM=LOC  COMPONENT=Y     BETWEEN  10.000     AND   10.000    
   KABS=  0.  TOLERANCE= 0.500000E-01

          1  NODES (OF          4  DEFINED) SELECTED BY  NSEL  COMMAND.
 >> Mapdl::InputFileS: Running command: 'ESLN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '

 SELECT      ALL ELEMENTS HAVING ANY NODE IN NODAL SET.

          1 ELEMENTS (OF          3  DEFINED) SELECTED FROM
        1 SELECTED NODES BY  ESLN COMMAND.
 >> Mapdl::InputFileS: Running command: 'FSUM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '

 ***** SUMMATION OF TOTAL    FORCES AND MOMENTS IN THE GLOBAL COORDINATE SYSTEM *****
  FX  =   0.000000    
  FY  =  -900.0000    
  FZ  =   0.000000    
  MX  =   0.000000    
  MY  =   0.000000    
  MZ  =   0.000000    

 SUMMATION POINT=  0.0000      0.0000      0.0000    
 >> Mapdl::InputFileS: Running command: '*GET,REAC_1,FSUM,,ITEM,FY                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      '

 *GET  REAC_1    FROM  FSUM  ITEM=ITEM FY    VALUE= -900.000000    
 >> Mapdl::InputFileS: Running command: 'NSEL,S,LOC,Y,0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 '

 SELECT       FOR ITEM=LOC  COMPONENT=Y     BETWEEN  0.0000     AND   0.0000    
   KABS=  0.  TOLERANCE= 0.100000E-05

          1  NODES (OF          4  DEFINED) SELECTED BY  NSEL  COMMAND.
 >> Mapdl::InputFileS: Running command: 'ESLN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '

 SELECT      ALL ELEMENTS HAVING ANY NODE IN NODAL SET.

          1 ELEMENTS (OF          3  DEFINED) SELECTED FROM
        1 SELECTED NODES BY  ESLN COMMAND.
 >> Mapdl::InputFileS: Running command: 'FSUM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '

 ***** SUMMATION OF TOTAL    FORCES AND MOMENTS IN THE GLOBAL COORDINATE SYSTEM *****
  FX  =   0.000000    
  FY  =  -600.0000    
  FZ  =   0.000000    
  MX  =   0.000000    
  MY  =   0.000000    
  MZ  =   0.000000    

 SUMMATION POINT=  0.0000      0.0000      0.0000    
 >> Mapdl::InputFileS: Running command: '*GET,REAC_2,FSUM,,ITEM,FY                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      '

 *GET  REAC_2    FROM  FSUM  ITEM=ITEM FY    VALUE= -600.000000    
 >> Mapdl::InputFileS: Running command: '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               '
 >> Mapdl::InputFileS: Running command: '*DIM,LABEL,CHAR,2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              '

 SET PARAMETER DIMENSIONS ON  LABEL TYPE=CHAR  DIMENSIONS=     2     1     1
 >> Mapdl::InputFileS: Running command: '*DIM,VALUE,,2,3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                '

 SET PARAMETER DIMENSIONS ON  VALUE TYPE=ARRA  DIMENSIONS=     2     3     1
 >> Mapdl::InputFileS: Running command: 'LABEL(1) = 'R1, lb','R2, lb '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  '

 PARAMETER LABEL(1) = R1, lb    R2, lb  
 >> Mapdl::InputFileS: Running command: '*VFILL,VALUE(1,1),DATA,900.0,600.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             '

 VECTOR FILL OPERATION  *VFILL  
  VALUE(1,1) = DATA
 VECTOR LENGTH=       2
 VALUES=  900.00      600.00    
 >> Mapdl::InputFileS: Running command: '*VFILL,VALUE(1,2),DATA,ABS(REAC_1),ABS(REAC_2)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 '

 VECTOR FILL OPERATION  *VFILL  
  VALUE(1,2) = DATA
 VECTOR LENGTH=       2
 VALUES=  900.00      600.00    
 >> Mapdl::InputFileS: Running command: '*VFILL,VALUE(1,3),DATA,ABS(REAC_1 / 900) ,ABS( REAC_2 / 600)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   '

 VECTOR FILL OPERATION  *VFILL  
  VALUE(1,3) = DATA
 VECTOR LENGTH=       2
 VALUES=  1.0000      1.0000    
 >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
 >> Mapdl::InputFileS: Running command: '/OUT,vm1,vrt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   '

 /OUTPUT FILE=         
 >> Mapdl::InputFileS: Running command: 'FINISH                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         '

 EXIT THE MAPDL POST1 DATABASE PROCESSOR


 ***** ROUTINE COMPLETED *****  CP =         0.000


 >> Mapdl::InputFileS: Running command: '*LIST,vm1,vrt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  '

  LISTING OF THE DATA ON FILE        
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '
  
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM,------------------- VM1 RESULTS COMPARISON ---------------------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          '
  ------------------- VM1 RESULTS COMPARISON ---------------------
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          '
  
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM,         |   TARGET   |   Mechanical APDL   |   RATIO                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     '
           |   TARGET   |   Mechanical APDL   |   RATIO
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          '
  
  >> Mapdl::InputFileS: Running command: '*VWRITE,LABEL(1),VALUE(1,1),VALUE(1,2),VALUE(1,3)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              '
  R1, lb          900.0       900.0   1.000
  R2, lb          600.0       600.0   1.000
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/COM,----------------------------------------------------------------                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          '
  ----------------------------------------------------------------
  >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
  >> Mapdl::InputFileS: Running command: '/OUT                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           '
 >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
 >> Mapdl::InputFileS: Running command: '/CLE,NOSTART                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   '

 CLEAR MAPDL DATABASE AND RESTART

 Ansys Mechanical Enterprise                       
 >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
 >> Mapdl::InputFileS: Running command: '/HEF                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             C^?  '
 >> Mapdl::InputFileS: if 'StrCmd[0] == '/''
 >> Mapdl::InputFileS: Running command: '/HEF                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             C^?  '
 >> Mapdl::InputFileS: Reading output file'
 >> Mapdl::InputFileS: Output file read'
 >> Mapdl::InputFileS: Waiting for thread to finish.'

 *** ERROR ***                           CP =       0.000   TIME= 00:00:00
 An unexpected error ( Segmentation Violation ) has occurred...  MAPDL   
 internal data has been corrupted.  MAPDL is unable to recover and will  
 terminate.  Previously saved files are unaffected.  Please send the     
 data leading to this operation to your technical support provider, as   
 this will allow ANSYS, Inc to improve the program.                      

 *** NOTE ***                            CP =       0.000   TIME= 00:00:00
 Additional information about signal generated errors can be found in    
 QSView -> Dept/Team Info -> Core ANSYS Team -> DEBUGGING.               

 Current ANSYS Traceback:
     ANSYS                           
     GRPC_Start                      
     gRPCMapdlStart                  
     ANSYS

+--------------------- M A P D L   S T A T I S T I C S ------------------------+

Release: 2024 R2            Build: 24.2BETA   Update: UP20240603   Platform: LINUX x64   
Using: DEVELOPMENT Build
Date Run: 07/26/2024   Time: 07:12     Process ID: 592654
Operating System: Red Hat Enterprise Linux release 9.4 (Plow)

Processor Model: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz

Compiler: Intel(R) Fortran Compiler Classic Version 2021.9  (Build: 20230302)
          Intel(R) C/C++ Compiler Classic Version 2021.9  (Build: 20230302)
          Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303
 
Number of processes requested           :    4
Number of threads per process requested :    1
Total number of cores requested         :    4 (Distributed Memory Parallel)               
MPI Type: INTELMPI
MPI Version: Intel(R) MPI Library 2021.11 for Linux* OS


GPU Acceleration: Not Requested

Job Name: file

  Core                Machine Name   Working Directory
 -----------------------------------------------------
     0        lyolinlst5.ansys.com   /home/gayuso/tmp
     1        lyolinlst5.ansys.com   /home/gayuso/tmp
     2        lyolinlst5.ansys.com   /home/gayuso/tmp
     3        lyolinlst5.ansys.com   /home/gayuso/tmp
 
Latency time from master to core     1 =    3.138 microseconds
Latency time from master to core     2 =    3.261 microseconds
Latency time from master to core     3 =    3.054 microseconds
 
Communication speed from master to core     1 =  8427.67 MB/sec
Communication speed from master to core     2 =  3983.21 MB/sec
Communication speed from master to core     3 =  3991.54 MB/sec

Total CPU time for main thread                    :        0.0 seconds
Total CPU time summed for all threads             :        6.9 seconds

Elapsed time spent obtaining a license            :        2.7 seconds
Elapsed time spent pre-processing model (/PREP7)  :        0.0 seconds
Elapsed time spent solution - preprocessing       :        0.0 seconds
Elapsed time spent computing solution             :        0.0 seconds
Elapsed time spent solution - postprocessing      :        0.0 seconds
Elapsed time spent post-processing model (/POST1) :        0.0 seconds
Elapsed time spent post-processing model (/POST26):        0.3 seconds
 
Equation solver used                              :            Sparse (symmetric)

+------------------ E N D   M A P D L   S T A T I S T I C S -------------------+


 *-----------------------------------------------------------------------------*
 |                                                                             |
 |                               RUN COMPLETED                                 |
 |                                                                             |
 |-----------------------------------------------------------------------------|
 |                                                                             |
 |  Ansys MAPDL 2024 R2         Build 24.2BETA     UP20240603    LINUX x64     |
 |                                                                             |
 |-----------------------------------------------------------------------------|
 |                                                                             |
 |  Database Requested(-db)  1024 MB     Scratch Memory Requested      1024 MB |
 |  Maximum Database Used       1 MB     Maximum Scratch Memory Used     46 MB |
 |                                                                             |
 |-----------------------------------------------------------------------------|
 |                                                                             |
 |        CP Time      (sec) =          6.894       Time  =  07:12:48          |
 |        Elapsed Time (sec) =         30.000       Date  =  07/26/2024        |
 |                                                                             |
 *-----------------------------------------------------------------------------*
I0726 07:12:48.718028046  592779 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718016111, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718052499  592782 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718045158, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718033900  592778 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718022570, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718036176  592776 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718023572, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718079224  592780 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718071051, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718123037  592781 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718112957, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718140697  592783 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718132668, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718146127  592774 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718137964, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718134345  592777 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718126596, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718201919  592772 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718193710, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.718958500  592775 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 718954756, clock_type: 1 }, reserved=(nil))
I0726 07:12:48.719952955  592773 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992369, tv_nsec: 719949156, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.818985531  592774 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 818975394, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.818989384  592782 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 818972646, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.818988743  592776 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 818979975, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819034032  592777 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819025706, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819059077  592772 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819048695, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819070779  592778 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819061633, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819081412  592779 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819073245, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819075831  592781 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819066262, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819116336  592780 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819107909, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819085525  592783 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819077489, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.819982367  592775 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 819977932, clock_type: 1 }, reserved=(nil))
I0726 07:12:49.820960490  592773 completion_queue.cc:965]              grpc_completion_queue_next(cq=0x194d800, deadline=gpr_timespec { tv_sec: 1721992370, tv_nsec: 820958128, clock_type: 1 }, reserved=(nil))
Abort(33) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 33) - process 0

@germa89
Copy link
Collaborator Author

germa89 commented Jul 31, 2024 •
edited
Loading

Now it doesn't seem to fail... reaching up to 2500 iterations, whereas before it was failing around 30-60. 🧐

Let's keep this stale for the moment.

@germa89
Copy link
Collaborator Author

germa89 commented Jul 31, 2024

I tested also v25.1.0 docker image and it does not reproduce the error.

@germa89
Copy link
Collaborator Author

germa89 commented Jul 31, 2024

I should check with SMP.

@germa89 germa89 mentioned this issue Aug 8, 2024
Closed
@germa89
Copy link
Collaborator Author

germa89 commented Aug 12, 2024 •
edited
Loading

Working on Windows

Most of the times, the letour model runs fine on Windows (32GB), only occupying 4 GB approx. But there is quite a bit disk activity, so maybe it is using the pagefile.

One crash

I caught a crash while running letour model. The event viewer shows the following trace:

Application error

Faulting application name: ANSYS.EXE, version: 22.2.0.0, time stamp: 0x6282b822
Faulting module name: ANSYS.EXE, version: 22.2.0.0, time stamp: 0x6282b822
Exception code: 0xc00000fd
Fault offset: 0x000000001588a248
Faulting process id: 0xAC68
Faulting application start time: 0x1DAECEDF6FCC00E
Faulting application path: C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ANSYS.EXE
Faulting module path: C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ANSYS.EXE
Report Id: 5840adb1-993e-4392-b18c-ee72adc2a9c3
Faulting package full name: 
Faulting package-relative application ID:

Windows error reportings

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: ANSYS.EXE
P2: 22.2.0.0
P3: 6282b822
P4: ANSYS.EXE
P5: 22.2.0.0
P6: 6282b822
P7: c00000fd
P8: 000000001588a248
P9: 
P10: 

Attached files:

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_ANSYS.EXE_de4cd253106be3f1245c7ecd51721159db13ef8_1ce45f46_889b0325-78a8-476f-97ba-1678f4ecd707

Analysis symbol: 
Rechecking for solution: 0
Report Id: 5840adb1-993e-4392-b18c-ee72adc2a9c3
Report Status: 4
Hashed bucket: 
Cab Guid: 0

and

Fault bucket 0, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: ANSYS.EXE
P2: 22.2.0.0
P3: 6282b822
P4: ANSYS.EXE
P5: 22.2.0.0
P6: 6282b822
P7: c00000fd
P8: 000000001588a248
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.98950b1b-574c-41d3-8ffc-70c0549a64b1.tmp.WERInternalMetadata.xml

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_ANSYS.EXE_de4cd253106be3f1245c7ecd51721159db13ef8_1ce45f46_889b0325-78a8-476f-97ba-1678f4ecd707

Analysis symbol: 
Rechecking for solution: 0
Report Id: 5840adb1-993e-4392-b18c-ee72adc2a9c3
Report Status: 268435456
Hashed bucket: 52920924216d799cccbf20d50d69eb4f
Cab Guid: 0

Report.wer

 .\Report.wer
Version=1
EventType=APPCRASH
EventTime=133643790489818361
ReportType=2
Consent=1
UploadTime=133643790500135082
ReportStatus=268435456
ReportIdentifier=5afe4407-1e7f-48c5-9f5f-5237faa88b20
IntegratorReportIdentifier=9b141bd1-af2f-4806-8cbc-7981721a8a3d
Wow64Host=34404
NsAppName=ANSYS.EXE
OriginalFilename=ANSYS.EXE
AppSessionGuid=00005244-0003-0006-5989-112651ccda01
TargetAppId=W:0006ef344ed275bf085ad72a71e82625532f00000904!0000935e8d9f0c708ab931c0d0578cdcc3a7c524fe70!ANSYS.exe
TargetAppVer=2022//05//16:20:46:26!1d8aa1c2!ANSYS.exe
BootId=4294967295
TargetAsId=100254
UserImpactVector=48
IsFatal=1
EtwNonCollectReason=1
Response.BucketId=213c47023fd5b97890d986368338f36b
Response.BucketTable=4
Response.LegacyBucketId=0
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=ANSYS.EXE
Sig[1].Name=Application Version
Sig[1].Value=22.2.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=6282b822
Sig[3].Name=Fault Module Name
Sig[3].Value=ANSYS.EXE
Sig[4].Name=Fault Module Version
Sig[4].Value=22.2.0.0
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=6282b822
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=000000000440370f
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.26058.2.0.0.256.4
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=5f36
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=5f36caf68c43ec5ec864144a162e33b9
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=afb5
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=afb500151318b1aae602565193e2903f
UI[2]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ANSYS.EXE
LoadedModule[0]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ANSYS.EXE
LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\WINDOWS\System32\KERNEL32.DLL
LoadedModule[3]=C:\WINDOWS\System32\KERNELBASE.dll
LoadedModule[4]=C:\WINDOWS\System32\PSAPI.DLL
LoadedModule[5]=C:\WINDOWS\System32\CRYPT32.dll
LoadedModule[6]=C:\WINDOWS\System32\ucrtbase.dll
LoadedModule[7]=C:\WINDOWS\SYSTEM32\IPHLPAPI.DLL
LoadedModule[8]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansMemManager.dll
LoadedModule[9]=C:\Program Files\ANSYS Inc\v222\ansys\bin\winx64\GPU\SIM\AnsGPU.dll
LoadedModule[10]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\MPI\IntelMPI\AnsMPI.dll
LoadedModule[11]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ans_gil.dll
LoadedModule[12]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\MPI\IntelMPI\ansScaLAPACK.dll
LoadedModule[13]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\cadoe_algorithms.dll
LoadedModule[14]=C:\WINDOWS\System32\ADVAPI32.dll
LoadedModule[15]=C:\WINDOWS\System32\msvcrt.dll
LoadedModule[16]=C:\WINDOWS\System32\sechost.dll
LoadedModule[17]=C:\WINDOWS\System32\bcrypt.dll
LoadedModule[18]=C:\WINDOWS\System32\RPCRT4.dll
LoadedModule[19]=C:\WINDOWS\System32\COMDLG32.dll
LoadedModule[20]=C:\WINDOWS\System32\msvcp_win.dll
LoadedModule[21]=C:\WINDOWS\System32\combase.dll
LoadedModule[22]=C:\WINDOWS\System32\shcore.dll
LoadedModule[23]=C:\WINDOWS\System32\USER32.dll
LoadedModule[24]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeKernel.dll
LoadedModule[25]=C:\WINDOWS\System32\win32u.dll
LoadedModule[26]=C:\WINDOWS\System32\GDI32.dll
LoadedModule[27]=C:\WINDOWS\System32\gdi32full.dll
LoadedModule[28]=C:\WINDOWS\System32\SHLWAPI.dll
LoadedModule[29]=C:\WINDOWS\System32\SHELL32.dll
LoadedModule[30]=C:\WINDOWS\System32\WS2_32.dll
LoadedModule[31]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeMath.dll
LoadedModule[32]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeLegacy.dll
LoadedModule[33]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeReaders.dll
LoadedModule[34]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansOpenMP.dll
LoadedModule[35]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeInterpolation.dll
LoadedModule[36]=C:\Program Files\ANSYS Inc\v222\tp\IntelCompiler\2019.5.281\winx64\libifcoremd.dll
LoadedModule[37]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\VCRUNTIME140.dll
LoadedModule[38]=C:\WINDOWS\System32\imagehlp.dll
LoadedModule[39]=C:\Program Files\ANSYS Inc\v222\tp\IntelMKL\2020.0.166\winx64\mkl_rt.dll
LoadedModule[40]=C:\Program Files\ANSYS Inc\v222\commonfiles\MPI\Intel\2021.5.0\winx64\bin\release\impi.dll
LoadedModule[41]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\CadoeReadersExt.dll
LoadedModule[42]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\mnflib.dll
LoadedModule[43]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\UserFcgLawLib.dll
LoadedModule[44]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansResourcePredict.dll
LoadedModule[45]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\libgmp-10.dll
LoadedModule[46]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansOpenSSL.dll
LoadedModule[47]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\MSVCP140.dll
LoadedModule[48]=C:\Program Files\ANSYS Inc\v222\tp\IntelCompiler\2019.5.281\winx64\libmmd.dll
LoadedModule[49]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\qhull_r.dll
LoadedModule[50]=C:\WINDOWS\SYSTEM32\GLU32.dll
LoadedModule[51]=C:\WINDOWS\SYSTEM32\AVIFIL32.dll
LoadedModule[52]=C:\WINDOWS\System32\ole32.dll
LoadedModule[53]=C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.26058.1000_none_d4508739c4aff23d\COMCTL32.dll
LoadedModule[54]=C:\WINDOWS\SYSTEM32\WINSPOOL.DRV
LoadedModule[55]=C:\WINDOWS\SYSTEM32\OPENGL32.dll
LoadedModule[56]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\libiomp5md.dll
LoadedModule[57]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\jpeg62.dll
LoadedModule[58]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\png.dll
LoadedModule[59]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\spooles.dll
LoadedModule[60]=C:\Program Files\ANSYS Inc\v222\tp\IntelCompiler\2019.5.281\winx64\libifportMD.dll
LoadedModule[61]=C:\Program Files\ANSYS Inc\v222\tp\IntelCompiler\2019.5.281\winx64\svml_dispmd.dll
LoadedModule[62]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansMETIS.dll
LoadedModule[63]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansPARMETIS.dll
LoadedModule[64]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\libtiff.dll
LoadedModule[65]=C:\WINDOWS\SYSTEM32\dbghelp.dll
LoadedModule[66]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\cgnsdll.dll
LoadedModule[67]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\dmumps.dll
LoadedModule[68]=C:\Program Files\ANSYS Inc\v222\ansys\bin\winx64\BLAS\INTEL\AnsBLAS.dll
LoadedModule[69]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\EXBlib.dll
LoadedModule[70]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\PrimeMesh.dll
LoadedModule[71]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\vtk.dll
LoadedModule[72]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\octree-mesh.dll
LoadedModule[73]=C:\WINDOWS\SYSTEM32\WSOCK32.dll
LoadedModule[74]=C:\WINDOWS\System32\OLEAUT32.dll
LoadedModule[75]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\libcrypto-1_1-x64.dll
LoadedModule[76]=C:\WINDOWS\SYSTEM32\MSVFW32.dll
LoadedModule[77]=C:\WINDOWS\SYSTEM32\WINMM.dll
LoadedModule[78]=C:\WINDOWS\SYSTEM32\cfgmgr32.dll
LoadedModule[79]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\zlib.dll
LoadedModule[80]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\hdf5.dll
LoadedModule[81]=C:\WINDOWS\SYSTEM32\MSACM32.dll
LoadedModule[82]=C:\WINDOWS\SYSTEM32\dxcore.dll
LoadedModule[83]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ansHDF.dll
LoadedModule[84]=C:\WINDOWS\SYSTEM32\dbgcore.DLL
LoadedModule[85]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\tg.dll
LoadedModule[86]=C:\WINDOWS\System32\IMM32.DLL
LoadedModule[87]=C:\WINDOWS\system32\napinsp.dll
LoadedModule[88]=C:\WINDOWS\System32\bcryptPrimitives.dll
LoadedModule[89]=C:\WINDOWS\System32\mswsock.dll
LoadedModule[90]=C:\WINDOWS\SYSTEM32\DNSAPI.dll
LoadedModule[91]=C:\WINDOWS\System32\NSI.dll
LoadedModule[92]=C:\WINDOWS\System32\winrnr.dll
LoadedModule[93]=C:\WINDOWS\system32\nlansp_c.dll
LoadedModule[94]=C:\WINDOWS\system32\wshbth.dll
LoadedModule[95]=C:\WINDOWS\SYSTEM32\dhcpcsvc6.DLL
LoadedModule[96]=C:\WINDOWS\SYSTEM32\dhcpcsvc.DLL
LoadedModule[97]=C:\WINDOWS\system32\uxtheme.dll
LoadedModule[98]=C:\WINDOWS\System32\MSCTF.dll
LoadedModule[99]=C:\WINDOWS\SYSTEM32\SspiCli.dll
LoadedModule[100]=C:\WINDOWS\System32\fwpuclnt.dll
LoadedModule[101]=C:\Windows\System32\rasadhlp.dll
LoadedModule[102]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ApipWrapper.dll
LoadedModule[103]=C:\WINDOWS\SYSTEM32\windows.storage.dll
LoadedModule[104]=C:\WINDOWS\SYSTEM32\kernel.appcore.dll
LoadedModule[105]=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\UAnBegLib.dll
LoadedModule[106]=C:\Program Files\ANSYS Inc\v222\tp\IntelMKL\2020.0.166\winx64\mkl_intel_thread.dll
LoadedModule[107]=C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
LoadedModule[108]=C:\Program Files\ANSYS Inc\v222\tp\IntelMKL\2020.0.166\winx64\mkl_core.dll
State[0].Key=Transport.DoneStage1
State[0].Value=1
OsInfo[0].Key=vermaj
OsInfo[0].Value=10
OsInfo[1].Key=vermin
OsInfo[1].Value=0
OsInfo[2].Key=verbld
OsInfo[2].Value=26058
OsInfo[3].Key=ubr
OsInfo[3].Value=1100
OsInfo[4].Key=versp
OsInfo[4].Value=0
OsInfo[5].Key=arch
OsInfo[5].Value=9
OsInfo[6].Key=lcid
OsInfo[6].Value=1033
OsInfo[7].Key=geoid
OsInfo[7].Value=217
OsInfo[8].Key=sku
OsInfo[8].Value=4
OsInfo[9].Key=domain
OsInfo[9].Value=1
OsInfo[10].Key=prodsuite
OsInfo[10].Value=256
OsInfo[11].Key=ntprodtype
OsInfo[11].Value=1
OsInfo[12].Key=platid
OsInfo[12].Value=10
OsInfo[13].Key=sr
OsInfo[13].Value=0
OsInfo[14].Key=tmsi
OsInfo[14].Value=222736800
OsInfo[15].Key=osinsty
OsInfo[15].Value=3
OsInfo[16].Key=iever
OsInfo[16].Value=11.1000.26058.0-11.0.1000
OsInfo[17].Key=portos
OsInfo[17].Value=0
OsInfo[18].Key=ram
OsInfo[18].Value=32487
OsInfo[19].Key=svolsz
OsInfo[19].Value=950
OsInfo[20].Key=wimbt
OsInfo[20].Value=0
OsInfo[21].Key=blddt
OsInfo[21].Value=240209
OsInfo[22].Key=bldtm
OsInfo[22].Value=1555
OsInfo[23].Key=bldbrch
OsInfo[23].Value=ge_release
OsInfo[24].Key=bldchk
OsInfo[24].Value=0
OsInfo[25].Key=wpvermaj
OsInfo[25].Value=0
OsInfo[26].Key=wpvermin
OsInfo[26].Value=0
OsInfo[27].Key=wpbuildmaj
OsInfo[27].Value=0
OsInfo[28].Key=wpbuildmin
OsInfo[28].Value=0
OsInfo[29].Key=osver
OsInfo[29].Value=10.0.26058.1100.amd64fre.ge_release.240209-1555
OsInfo[30].Key=buildflightid
OsInfo[30].Value=F5822673-3023-40E5-B154-CB65E5CC868A.1
OsInfo[31].Key=edition
OsInfo[31].Value=Enterprise
OsInfo[32].Key=ring
OsInfo[32].Value=External
OsInfo[33].Key=flightbranch
OsInfo[33].Value=Dev
OsInfo[34].Key=expid
OsInfo[34].Value=NG:1ABFE1,DR:11FFEC,DR:1200A8,DR:12042E,DR:129B2E,DR:12BE2B,DR:E55A4,DR:E756A,DR:E75FA,DR:EA07B,FX:114145DF,FX:124117A5,FX:12816BC3,FX:12EA15FA,FX:12EA4D41,FX:12ED0295,FX:12ED430B,FX:12F265E0,FX:12F47542,FX:12F98E44,FX:12FD5D07,FX:12FD6307,FX:12FDA634,FX:12FE5FF0,FX:1305C615,FX:13083122,FX:1309D046,FX:130D2A66,FX:1312FF8D,FX:13134674,FX:13136555,FX:13151AD4,FX:1316F9BF,FX:1318CA30,FX:1318CAEE,FX:1318CAEF,FX:1318CBED,FX:1318CBF1,FX:309F6CD,DR:CECBD,MD:283BAEF,ME:28279A6,ME:2A69053,ME:255521A
OsInfo[35].Key=fconid
OsInfo[35].Value=19193768,0,2,0;19193792,0,2,0;19194150,0,2,0;21031911,0,2,1;35681102,0,1,0;38142198,0,2,0;39145991,0,2,0;39146010,0,2,0;39263329,1,2,1;39281392,0,2,0;41267076,0,2,1;41655236,0,2,0;42105254,0,2,0;42187503,0,1,0;44470355,0,2,0;44776738,0,2,1;47268464,1,2,1;47370305,0,2,1;47402621,0,2,0;47557358,0,1,1;48126879,0,2,1;48531596,0,2,0;48559195,0,2,1;48635704,0,2,1;48662857,0,2,0;48742963,0,2,1;48818784,0,0,1;49199869,0,2,0;49204591,0,1,0;49239926,0,1,0;49309588,0,1,0;49577493,0,1,0;49615603,0,1,0;50039410,0,2,1;50657847,0,2,1;50762615,0,1,0
OsInfo[36].Key=containerid
OsInfo[37].Key=containertype
OsInfo[38].Key=edu
OsInfo[38].Value=0
OsInfo[39].Key=servicinginprogress
OsInfo[39].Value=0
OsInfo[40].Key=featureupdatependingreboot
OsInfo[40].Value=0
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Mechanical APDL Program
AppPath=C:\Program Files\ANSYS Inc\v222\ANSYS\bin\winx64\ANSYS.EXE
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=6D95CFA5907A9986222A3075999A824A
MetadataHash=-576973452

From the report.wer:

Sig[6].Name=Exception Code
Sig[6].Value=c0000005

From 1

A C0000005 error is memory error. Specifically, a C0000005 error is an access violation error caused by a buffer overrun

@germa89
Copy link
Collaborator Author

germa89 commented Aug 12, 2024

Running the vm1 on Windows I end up getting:

Output

...
 CLEAR MAPDL DATABASE AND RESTART

 Ansys Mechanical Enterprise                       

 *** ERROR ***                           CP =       0.000   TIME= 00:00:00
 An unexpected error ( SIG$ABORT ) has occurred...  ANSYS internal data  
 has been corrupted.  ANSYS is unable to recover and will terminate.     
 Previously saved files are unaffected.  Please send the data leading    
 to this operation to your technical support provider, as this will      
 allow ANSYS, Inc to improve the program.                                

 Current ANSYS Traceback:
     ANSYS                           
     GRPC_Start                      
     gRPCMapdlStart                  
     ANSYS

+--------------------- M A P D L   S T A T I S T I C S ------------------------+

Release: 2022 R2            Build: 22.2       Update: UP20220516   Platform: WINDOWS x64 
Date Run: 08/12/2024   Time: 22:09     Process ID: 32980
Operating System: Windows 10  (Build: 26058)
...

Crash file

Dumping the crash data for the current analysis.
    20: ELMATTRACK - 0x7892B330
    19: ELMATTRACK - 0x7892B330
    18: raise - 0xAADAE200
    17: abort - 0xAAD90E10
    16: terminate - 0xAADAE410
    15: _C_specific_handler_noexcept - 0xAAD9D8C0
    14: _chkstk - 0xAD3C2CD0
    13: RtlVirtualUnwind - 0xAD2913C0
    12: RtlRaiseException - 0xAD292EE0
    11: RaiseException - 0xAA878DB0
    10: CxxThrowException - 0xD5754760
    9: CertGetCertificateContextProperty - 0x8DFC19A4
    8: CertGetCertificateContextProperty - 0x8DFC19A4
    7: ELMATTRACK - 0x7892B330
    6: ELMATTRACK - 0x7892B330
    5: ELMATTRACK - 0x7892B330
    4: __lc_collate_cp_func - 0xAAD69E00
    3: BaseThreadInitThunk - 0xAC2B8D00
    2: RtlUserThreadStart - 0xAD2E7430

PyMAPDL output

This is shown in PyMAPDL as:

205
206
207
208
209
210
211
Traceback (most recent call last):
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\errors.py", line 310, in wrapper
    out = func(*args, **kwargs)
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 1843, in input
    _ = [chunk.cmdout for chunk in chunks]  # unstable
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 1843, in <listcomp>
    _ = [chunk.cmdout for chunk in chunks]  # unstable
  File "C:\Users\user\pymapdl\.venv_win\lib\site-packages\grpc\_channel.py", line 543, in __next__
    return self._next()
  File "C:\Users\user\pymapdl\.venv_win\lib\site-packages\grpc\_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Connection reset"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:50052 {created_time:"2024-08-12T20:09:29.6578424+00:00", grpc_status:14, grpc_message:"Connection reset"}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\pymapdl\tmp\crash\main.py", line 17, in <module>
    with mapdl.non_interactive:
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\mapdl_core.py", line 1379, in __exit__
    self._parent()._flush_stored()
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 2012, in _flush_stored
    out = self.input(
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\errors.py", line 328, in wrapper
    handle_generic_grpc_error(error, func, args, kwargs)
  File "C:\Users\user\pymapdl\src\ansys\mapdl\core\errors.py", line 381, in handle_generic_grpc_error       
    raise MapdlExitedError(msg)
ansys.mapdl.core.errors.MapdlExitedError: Error:
MAPDL server connection terminated unexpectedly while running:
  C:\Users\user\AppData\Local\Temp\tmp_wigoebqisy.inp
called by:
  input

Error:
  Connection reset
Full error:
<_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Connection reset"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:50052 {created_time:"2024-08-12T20:09:29.6578424+00:00", grpc_status:14, grpc_message:"Connection reset"}"
>

@germa89
Copy link
Collaborator Author

germa89 commented Aug 14, 2024

Analysing the dump file (from: #3313 (comment)) on Windbg, it seems we hit a break point?

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

Full dump analysis

Details 

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 2624

    Key  : Analysis.Elapsed.mSec
    Value: 62272

    Key  : Analysis.IO.Other.Mb
    Value: 16

    Key  : Analysis.IO.Read.Mb
    Value: 1

    Key  : Analysis.IO.Write.Mb
    Value: 29

    Key  : Analysis.Init.CPU.mSec
    Value: 624

    Key  : Analysis.Init.Elapsed.mSec
    Value: 4519

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 609

    Key  : Failure.Bucket
    Value: BREAKPOINT_80000003_ANSYS.exe!Unknown

    Key  : Failure.Hash
    Value: {36550f4e-cd90-8ab9-e27b-e69494726600}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 3587071

    Key  : Timeline.Process.Start.DeltaSec
    Value: 160

    Key  : WER.OS.Branch
    Value: ge_release

    Key  : WER.OS.Version
    Value: 10.0.26058.1000

    Key  : WER.Process.Version
    Value: 22.2.0.0


FILE_IN_CAB:  file.dmp

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  00004cc4

PROCESS_NAME:  ANSYS.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
000000c5`265f8748 00007ffd`ad2e54db     : 000002b4`3a0c0960 00000000`00000000 000002b4`3a0c0950 000000c5`265f8770 : ntdll!NtWaitForAlertByThreadId+0x14
000000c5`265f8750 00007ffd`aa887077     : 00000000`00000000 000000c5`265f8930 000002b4`39e0c8c8 ffffffff`fffffffe : ntdll!RtlSleepConditionVariableCS+0x10b
000000c5`265f87c0 00007ff7`8e31e4c2     : 000002b4`39e0c870 00000000`0000000f 00007ffc`d31dede0 00000000`00000000 : KERNELBASE!SleepConditionVariableCS+0x27
000000c5`265f87f0 00007ff7`8e3046a0     : 000000c5`265f88f0 000002b4`39f34940 000002b4`00000000 00000000`000000ac : ANSYS!CertGetCertificateContextProperty+0x35cb1e
000000c5`265f8850 00007ff7`8e2ff7ff     : 000000c5`265f88f0 000002b4`39f34940 00000000`00000000 000000c5`265f9908 : ANSYS!CertGetCertificateContextProperty+0x342cfc
000000c5`265f8890 00007ff7`8d6904b3     : 000000c5`265f8968 000000c5`265f88f0 000000c5`265f9290 000002b4`39de3f70 : ANSYS!CertGetCertificateContextProperty+0x33de5b
000000c5`265f88f0 00007ff7`8d68fd53     : 000000c5`265f9348 000000c5`265f9368 000000c5`265f9388 00007ffd`aad1008b : ANSYS!ELMATTRACK+0x14d65183
000000c5`265f92d0 00007ff7`8d68e847     : 000000c5`265f96f0 00007ff7`8f2ffd34 ffffffff`ffffffff 81010101`01010100 : ANSYS!ELMATTRACK+0x14d64a23
000000c5`265f93e0 00007ff7`8485812d     : 00000000`00000000 00000000`00000000 00000000`00000000 000000c5`265f99a0 : ANSYS!ELMATTRACK+0x14d63517
000000c5`265f9960 00007ff7`7bb15504     : 000000c5`265fe4b9 000000c5`265fd524 000000c5`265fa4a8 00000000`00000005 : ANSYS!ELMATTRACK+0xbf2cdfd
000000c5`265f9ad0 00007ff7`7895fb8e     : 000002b4`3883decd 00007ffd`ac2e64d0 00000000`00000000 00000000`00000000 : ANSYS!ELMATTRACK+0x31ea1d4
000000c5`265fe7c0 00007ff7`78955be7     : 00007ff7`78890000 00000000`00000000 000002b4`3883decd 00007ffd`0000000a : ANSYS!ELMATTRACK+0x3485e
000000c5`265ff620 00007ff7`8e11a00a     : 00007ff7`78890000 00000000`00000000 000002b4`3883decd 00000000`0000000a : ANSYS!ELMATTRACK+0x2a8b7
000000c5`265ff760 00007ffd`ac2b8d17     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ANSYS!CertGetCertificateContextProperty+0x158666
000000c5`265ff7a0 00007ffd`ad2e7450     : 00000000`00000000 00000000`00000000 000004f0`fffffb30 000004d0`fffffb30 : kernel32!BaseThreadInitThunk+0x17
000000c5`265ff7d0 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x20


STACK_COMMAND:  ~0s; .ecxr ; kb

SYMBOL_NAME:  ansys+35cb1e

MODULE_NAME: ANSYS

IMAGE_NAME:  ANSYS.exe

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_ANSYS.exe!Unknown

OS_VERSION:  10.0.26058.1000

BUILDLAB_STR:  ge_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  22.2.0.0

FAILURE_ID_HASH:  {36550f4e-cd90-8ab9-e27b-e69494726600}

Followup:     MachineOwner

@germa89
Copy link
Collaborator Author

germa89 commented Aug 14, 2024 •
edited
Loading

While running v241 I saw it crashing:

PyMAPDL output

...
12
13
14
Traceback (most recent call last):
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\errors.py", line 310, in wrapper
    out = func(*args, **kwargs)
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 1843, in input
    _ = [chunk.cmdout for chunk in chunks]  # unstable
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 1843, in <listcomp>
    _ = [chunk.cmdout for chunk in chunks]  # unstable
  File "C:\Users\gayuso\pymapdl\.venv_win\lib\site-packages\grpc\_channel.py", line 543, in __next__
    return self._next()
  File "C:\Users\gayuso\pymapdl\.venv_win\lib\site-packages\grpc\_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Connection reset"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:10.105.52.42:50052 {created_time:"2024-08-14T17:37:11.172502+00:00", grpc_status:14, grpc_message:"Connection reset"}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\gayuso\pymapdl\tmp\crash\main.py", line 18, in <module>
    with mapdl.non_interactive:
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\mapdl_core.py", line 1379, in __exit__
    self._parent()._flush_stored()
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\mapdl_grpc.py", line 2012, in _flush_stored
    out = self.input(
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\errors.py", line 328, in wrapper
    handle_generic_grpc_error(error, func, args, kwargs)
  File "C:\Users\gayuso\pymapdl\src\ansys\mapdl\core\errors.py", line 381, in handle_generic_grpc_error
    raise MapdlExitedError(msg)
ansys.mapdl.core.errors.MapdlExitedError: Error:
MAPDL server connection terminated unexpectedly while running:
  C:\Users\gayuso\AppData\Local\Temp\tmp_jrnyiitobl.inp
called by:
  input

Error:
  Connection reset
Full error:
<_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Connection reset"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:10.105.52.42:50052 {created_time:"2024-08-14T17:37:11.172502+00:00", grpc_status:14, grpc_message:"Connection reset"}"

MAPDL output file

Process died.

 CLEAR MAPDL DATABASE AND RESTART

 Ansys Mechanical Enterprise                       

 *** ERROR ***                           CP =       0.000   TIME= 00:00:00
 An unexpected error ( SIG$ABORT ) has occurred...  MAPDL internal data  
 has been corrupted.  MAPDL is unable to recover and will terminate.     
 Previously saved files are unaffected.  Please send the data leading    
 to this operation to your technical support provider, as this will      
 allow ANSYS, Inc to improve the program.                                

 Current ANSYS Traceback:
     ANSYS                           
     GRPC_Start                      
     gRPCMapdlStart                  
     ANSYS

Event viewer

Log Name:      Application
Source:        Windows Error Reporting
Date:          8/14/2024 7:34:37 PM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      ottw1100701.win.ansys.com
Description:
Fault bucket , type 0
Event Name: RADAR_PRE_LEAK_64
Response: Not available
Cab Id: 0

Problem signature:
P1: ANSYS.exe
P2: 24.1.0.0
P3: 10.0.22000.2.0.0
P4: 
P5: 
P6: 
P7: 
P8: 
P9: 
P10: 

Attached files:
\\?\C:\Users\gayuso\AppData\Local\Temp\RDR4F13.tmp\empty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.2c8ea786-4606-4e96-8fb2-44aa4e324cc1.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.47a34cb2-2e75-4c7d-a590-976593edbd0b.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.7172588c-0f44-4a46-a1e9-bf83c7b685cf.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.215ac5a5-584b-477b-8749-c6b249c78ddf.tmp.txt

These files may be available here:


Analysis symbol: 
Rechecking for solution: 0
Report Id: e79fec3c-bfb8-4fc7-80a8-88587bb236cc
Report Status: 536870912
Hashed bucket: 
Cab Guid: 0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2024-08-14T17:34:37.9651330Z" />
    <EventRecordID>106089</EventRecordID>
    <Correlation />
    <Execution ProcessID="11180" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>ottw1100701.win.ansys.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>RADAR_PRE_LEAK_64</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>ANSYS.exe</Data>
    <Data>24.1.0.0</Data>
    <Data>10.0.22000.2.0.0</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
\\?\C:\Users\gayuso\AppData\Local\Temp\RDR4F13.tmp\empty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.2c8ea786-4606-4e96-8fb2-44aa4e324cc1.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.47a34cb2-2e75-4c7d-a590-976593edbd0b.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.7172588c-0f44-4a46-a1e9-bf83c7b685cf.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.215ac5a5-584b-477b-8749-c6b249c78ddf.tmp.txt</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>e79fec3c-bfb8-4fc7-80a8-88587bb236cc</Data>
    <Data>536870912</Data>
    <Data>
    </Data>
    <Data>0</Data>
  </EventData>
</Event>

From this event, it seems it might be a memory leak issue reference

Task manager

CPUs

image

Memory RAM

image

Dump file analysis:

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Unable to verify timestamp for libiomp5md.dll
*** WARNING: Unable to verify timestamp for msvcp140.dll

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1202

    Key  : Analysis.Elapsed.mSec
    Value: 45795

    Key  : Analysis.IO.Other.Mb
    Value: 22

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 40

    Key  : Analysis.Init.CPU.mSec
    Value: 577

    Key  : Analysis.Init.Elapsed.mSec
    Value: 28767

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 106

    Key  : Failure.Bucket
    Value: BREAKPOINT_80000003_ANSYS.exe!Unknown

    Key  : Failure.Hash
    Value: {36550f4e-cd90-8ab9-e27b-e69494726600}

    Key  : Timeline.Process.Start.DeltaSec
    Value: 412

    Key  : WER.OS.Branch
    Value: co_release

    Key  : WER.OS.Version
    Value: 10.0.22000.1

    Key  : WER.Process.Version
    Value: 5.0.1.0


FILE_IN_CAB:  file.dmp

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  000021a8

PROCESS_NAME:  ANSYS.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
00000056`5c5f9198 00007ffc`c2dab087     : 00007ff7`c5a295c8 00000000`00000173 00007ff7`c5a295d0 00007ff7`c5a295d0 : ntdll!NtWaitForAlertByThreadId+0x14
00000056`5c5f91a0 00007ffc`c07dde09     : 00000000`00000000 00000188`f5bca818 00000189`85ae0940 00000000`00000048 : ntdll!RtlSleepConditionVariableSRW+0x137
00000056`5c5f9220 00007ff7`c2e597a4     : 00000000`00000000 00000000`00000000 00000056`5c5f9310 00000189`86f383b0 : KERNELBASE!SleepConditionVariableSRW+0x29
00000056`5c5f9260 00000000`00000000     : 00000000`00000000 00000056`5c5f9310 00000189`86f383b0 00000000`00000000 : ANSYS+0xfbb97a4


STACK_COMMAND:  ~0s; .ecxr ; kb

SYMBOL_NAME:  ansys+fbb97a4

MODULE_NAME: ANSYS

IMAGE_NAME:  ANSYS.exe

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_ANSYS.exe!Unknown

OS_VERSION:  10.0.22000.1

BUILDLAB_STR:  co_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  24.1.0.0

FAILURE_ID_HASH:  {36550f4e-cd90-8ab9-e27b-e69494726600}

Again the breakpoint error...

@germa89
Copy link
Collaborator Author

germa89 commented Aug 20, 2024

For the record, crashing errors are not logged to thread output files (file1.out, file2.out, file3.out, ...)

I mean crashing errors like:

 CLEAR MAPDL DATABASE AND RESTART

 Ansys Mechanical Enterprise                       

 *** ERROR ***                           CP =       0.000   TIME= 00:00:00
 An unexpected error ( SIG$ABORT ) has occurred...  MAPDL internal data  
 has been corrupted.  MAPDL is unable to recover and will terminate.     
 Previously saved files are unaffected.  Please send the data leading    
 to this operation to your technical support provider, as this will      
 allow ANSYS, Inc to improve the program.                                

 Current ANSYS Traceback:
     ANSYS                           
     GRPC_Start                      
     gRPCMapdlStart                  
     ANSYS

@germa89 germa89 mentioned this issue Oct 31, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@germa89 germa89

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@germa89