Agent leaves hanging child processes

[mssalvatore]

Description

After the agent completes its task, it is unable to exit. It seems that the process is being kept open by its child processes. It's unclear why the child processes do not exit. This occurs intermittently but happens most frequently if the agent is run within a docker container.

The effects of this are:

1. The agent leaves processes hanging around and consuming some small amount of resources
2. Other agents cannot start on the machine since the SystemSingleton is never properly released. It would seem that maybe one of the child processes has a copy of the file descriptor for the unix socket, but it's not quite clear how that's possible since this occurs even of the child processes are spawned before the socket is opened.

Tasks

• Fix it (0.5d) @cakekoa
  • Rebase on develop
  • Run ETE tests

[mssalvatore]

It turns out the root cause was the Log4Shell exploiter's LDAP server was not being properly closed. This is resolved by 69c2c88.

EDIT:
This still isn't resolved

2023-01-12 03:02:34,039 [115:ExploiterThread-04:DEBUG] ldap_server.stop.198: Stopping LDAP exploit server
2023-01-12 03:02:34,039 [211:ExploiterThread-04:INFO] _legacy.publishToNewObserver.147: Received SIGTERM, shutting down.
2023-01-12 03:02:34,040 [211:ExploiterThread-04:INFO] _legacy.publishToNewObserver.147: (TCP Port 8008 Closed)
2023-01-12 03:02:34,041 [211:ExploiterThread-04:INFO] _legacy.publishToNewObserver.147: Stopping factory <infection_monkey.exploit.log4shell_utils.ldap_server.LDAPServerFactory object at 0x7fa8b80bf7d0>
2023-01-12 03:02:34,041 [211:ExploiterThread-04:INFO] _legacy.publishToNewObserver.147: Main loop terminated.
2023-01-12 03:02:37,000 [115:AgentEventForwarder:DEBUG] agent_event_forwarder.flush.94: Sending 3 Agent events to the Island: [PropagationEvent(source=UUID('d2fbc46f-72d2-44dc-8341-460f77b75f3b'), target=IPv4Address('10.2.3.46'), timestamp=1673492534.8145223, tags=frozenset({'log4shell-exploiter', 'attack-t1203', 'attack-t1105'}), success=True, exploiter_name='Log4ShellExploiter', error_message=''), ExploitationEvent(source=UUID('d2fbc46f-72d2-44dc-8341-460f77b75f3b'), target=IPv4Address('10.2.3.46'), timestamp=1673492553.9918516, tags=frozenset({'log4shell-exploiter', 'attack-t1203', 'attack-t1110'}), success=True, exploiter_name='Log4ShellExploiter', error_message=''), PropagationEvent(source=UUID('d2fbc46f-72d2-44dc-8341-460f77b75f3b'), target=IPv4Address('10.2.3.46'), timestamp=1673492534.8145223, tags=frozenset({'log4shell-exploiter', 'attack-t1203', 'attack-t1105'}), success=True, exploiter_name='Log4ShellExploiter', error_message='')]
2023-01-12 03:02:37,001 [115:AgentEventForwarder:DEBUG] http_client._send_request.132: POST https://10.2.2.250:5000/api/agent-events, timeout=5
2023-01-12 03:02:37,068 [115:AutomatedMasterThread:DEBUG] http_client._send_request.132: GET https://10.2.2.250:5000/api/agent-signals/d2fbc46f-72d2-44dc-8341-460f77b75f3b, timeout=2.5
2023-01-12 03:02:37,884 [115:TCPConnectionHandler:DEBUG] tcp_connection_handler.run.42: New connection received from: ('10.2.2.12', 39970)
2023-01-12 03:02:37,886 [115:TCPConnectionHandler:DEBUG] relay_user_handler.add_relay_user.64: Added relay user RelayUser(address='10.2.2.12', time_remaining=0.000)
2023-01-12 03:02:37,899 [115:SocketsPipeThread-2:DEBUG] tcp_pipe_spawner._handle_pipe_closed.58: Closing pipe <SocketsPipe(SocketsPipeThread-2, started daemon 140362366555904)>
2023-01-12 03:02:42,078 [115:AutomatedMasterThread:DEBUG] http_client._send_request.132: GET https://10.2.2.250:5000/api/agent-signals/d2fbc46f-72d2-44dc-8341-460f77b75f3b, timeout=2.5
2023-01-12 03:02:42,924 [115:TCPConnectionHandler:DEBUG] tcp_connection_handler.run.42: New connection received from: ('10.2.2.12', 39976)
2023-01-12 03:02:42,925 [115:TCPConnectionHandler:DEBUG] relay_user_handler.disconnect_user.88: Disconnected user 10.2.2.12
2023-01-12 03:02:46,893 [115:AgentHeart:DEBUG] http_client._send_request.132: POST https://10.2.2.250:5000/api/agent/d2fbc46f-72d2-44dc-8341-460f77b75f3b/heartbeat, timeout=5
2023-01-12 03:02:47,089 [115:AutomatedMasterThread:DEBUG] http_client._send_request.132: GET https://10.2.2.250:5000/api/agent-signals/d2fbc46f-72d2-44dc-8341-460f77b75f3b, timeout=2.5
2023-01-12 03:02:49,053 [115:ExploiterThread-04:WARNING] ldap_server.stop.206: Timed out while waiting for the LDAP exploit server to stop