Unreachable Peer PLC issue since 4.0.7

[patrickjaigner]

First occurrence: 14.02.2023,14:47:40 on Mb02
It only happens on 3/4 of stations. Me06 is currently in maintenance and not deployed.

Mb02:

Mc04:

Md05:

Traceback (most recent call last):

File "C:\Users\EnclosureMc04\Documents\pyra\pyra-4.0.7\packages\core\main.py", line 116, in run
m.run(config)

File "C:\Users\EnclosureMc04\Documents\pyra\pyra-4.0.7\packages\core\modules\enclosure_control.py", line 91, in run
self.plc_interface.connect()

File "C:\Users\EnclosureMc04\Documents\pyra\pyra-4.0.7\packages\core\interfaces\plc_interface.py", line 140, in connect
self.plc.connect(self.plc_ip, 0, 1)

File "C:\Users\EnclosureMc04\AppData\Local\Programs\Python\Python310\lib\site-packages\snap7\client_init_.py", line 24, in f
check_error(code, context="client")

File "C:\Users\EnclosureMc04\AppData\Local\Programs\Python\Python310\lib\site-packages\snap7\common.py", line 89, in check_error
raise RuntimeError(error)

RuntimeError: b' TCP : Unreachable peer'

[dostuffthatmatters]

This is probably due to 4.0.6 using python-snap7@1.1 and 4.0.7 using python-snap7@1.3. This happened because we do automatic library upgrades with backwards compatible releases (here: ^1.1).

Fix options:

1. Fix the version at 1.1.
2. Stay at ^1.1 and add resilience to the enclosure control procedure -> only raise an exception when the connection could not be established for X times in a row.

I prefer option 2, and we already implemented this resilience built into the PLC communication:

pyra/packages/core/modules/enclosure_control.py

Lines 158 to 172 in 53e987f

	# Allows for PLC connection downtime of 10 minutes before an error is raised.
	# Allows PLC connection to heal itself.
	except Snap7Exception as e:
	logger.exception(e)
	now = time.time()
	seconds_since_error_occured = now - self.last_plc_connection_time
	if seconds_since_error_occured > 600:
	raise interfaces.PLCInterface.PLCError(
	"Snap7Exception persisting for 10+ minutes"
	)
	else:
	logger.info(
	f"Snap7Exception persisting for {round(seconds_since_error_occured/60, 2)}"
	+ " minutes. Sending email at 10 minutes."
	)

So, why do we now get a RuntimeError instead of a Snap7Exception? Because they changed it. Here is the 1.1 implementation of the error-raising function:

https://github.com/gijzelaerr/python-snap7/blob/3279ea1ca96b101021cb12890c929296e4724253/snap7/common.py#L67-L75

... and here is the 1.3 implementation:

https://github.com/gijzelaerr/python-snap7/blob/d26cfbe7da002c5527f5a1de549cc4c728640908/snap7/common.py#L75-L89

My change proposal:

 -  except Snap7Exception as e:
 +  except (Snap7Exception, RuntimeError) as e:

[dostuffthatmatters]

In my opinion, this change in the python-snap7 library is a breaking one because it is not a bug but an intentional change in behavior. Hence, it should be a 2.0 (https://semver.org/).

[patrickjaigner]

Good analysis. We should discuss if backwards compatible library updates of dependencies should be done automatically for our systems.

I would strongly advice against it.
There are always risks for the instrument we can’t fully access and we don‘t have a indoor hardware setup to perform long duration tests for low occurrence errors.