model replay random failures

[adeebshihadeh]

Should be easy to catch running a loop. @deanlee want to take a look at and see why this isn't fully deterministic? There's a chance it's a test bug too.

https://jenkins.comma.life/blue/organizations/jenkins/openpilot/detail/master/1233/pipeline

Logs attached from that failed jenkins run.
log.txt

[deanlee]

Is it possible that it is caused by the occasional difference in the update order of 'lateralPlan,liveCalibration' and vipc frame in modeld?

openpilot/selfdrive/test/process_replay/model_replay.py

Lines 97 to 106 in a112599

	for msg in tqdm(lr_list):
	if msg.which() == "liveCalibration":
	last_calib = list(msg.liveCalibration.rpyCalib)
	pm.send(msg.which(), replace_calib(msg, last_calib))
	elif msg.which() == "lateralPlan":
	last_desire = msg.lateralPlan.desire
	elif msg.which() in ["roadCameraState", "driverCameraState"]:
	ret = process_frame(msg, pm, sm, log_msgs, vipc_server, spinner, frs, frame_idxs, last_desire)
	if ret is None:
	break

If this is the case, waiting for a while after send msg, or update live calibration in main thread to ensuring the order of updates in modeld may solve this problem

[adeebshihadeh]

hopefully fixed by #23237