Connection Issues and unresponsive panel buttons

[augalos]

Hi,

I am having a hard time making aqualinkd to work with my RS6 panel. I get some checksum errors and incomplete information with the serial logger:

AqualinkD serial_logger V2.2
Notice: RS Serial: Port /dev/ttyUSB0 low latency mode is set
Notice: RS Serial: Logging serial information!
Please wait.|

Notice: RS Serial: RS485 interface received 600 packets in 13 seconds (~46.15 Msg/Sec)
Notice: RS Serial: Jandy Control Panel Model :
Notice: RS Serial: Jandy Control Panel Version :
Notice: RS Serial: Jandy ID's found
Notice: RS Serial: ID 0x08 is in use <-- RS Keypad
Notice: RS Serial: ID 0x09 is not used <-- can use for Aqualinkd
Notice: RS Serial: ID 0x0a is not used <-- can use for Aqualinkd
Notice: RS Serial: ID 0x0b is not used <-- can use for Aqualinkd
Notice: RS Serial:

Notice: RS Serial: Suggested aqualinkd.conf values
Notice: RS Serial: -------------------------
Notice: RS Serial: device_id = 0x09
Notice: RS Serial: -------------------------

Log output:

Aug 13 00:43:27 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:27 raspberrypi aqualinkd[706]: RS Serial: Serial read bad Jandy checksum, ignoring
Aug 13 00:43:27 raspberrypi aqualinkd[706]: RS Serial: Read Jandy packet BAD PACKET To 0xff of type Probe | HEX: 0x10|0x02|0xff|0x00|0x10|0x02|0x08|0x02|0x00|0x00|0x00|0x00|0x00|0x1c|0x10|0x03|
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:30 raspberrypi aqualinkd[706]: RS Serial: Serial read bad Jandy checksum, ignoring
Aug 13 00:43:30 raspberrypi aqualinkd[706]: RS Serial: Read Jandy packet BAD PACKET To 0x10 of type Probe | HEX: 0x10|0x02|0x10|0x00|0x22|0x10|0xff|0x00|0x10|0x02|0x08|0x02|0x00|0x00|0x00|0x00|0x00|0x1c|0x10|0x03|
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:43 raspberrypi aqualinkd[706]: RS Serial: Serial read bad Jandy checksum, ignoring
Aug 13 00:43:43 raspberrypi aqualinkd[706]: RS Serial: Read Jandy packet BAD PACKET To 0xf0 of type Probe | HEX: 0x10|0x02|0xf0|0x00|0x10|0x02|0x08|0x02|0x00|0x00|0x00|0x00|0x00|0x1c|0x10|0x03|
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:45 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:47 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:43:48 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:43:48 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:43:48 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: RS Message :- '6521 REV E'
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel version 6521 REV E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: AllButton: Control Panel revision E
Aug 13 00:44:11 raspberrypi aqualinkd[706]: RS Serial: Serial read bad Jandy checksum, ignoring
Aug 13 00:44:11 raspberrypi aqualinkd[706]: RS Serial: Read Jandy packet BAD PACKET To 0x10 of type Probe | HEX: 0x10|0x02|0x10|0x00|0x22|0x10|0xff|0x00|0x10|0x02|0x08|0x02|0x00|0x00|0x00|0x00|0x00|0x1c|0x10|0x03|

Aug 13 00:59:03 raspberrypi aqualinkd[753]: AqualinkD: Config BTN Spa_Heater = label Spa Heater |
Aug 13 00:59:03 raspberrypi aqualinkd[753]: AqualinkD: Config BTN Solar_Heater = label Solar Heater |
Aug 13 00:59:03 raspberrypi aqualinkd[754]: NetService:Starting network services thread
Aug 13 00:59:03 raspberrypi aqualinkd[754]: NetService:Starting web server on port 80
Aug 13 00:59:03 raspberrypi aqualinkd[754]: NetService:Starting MQTT client to localhost:1884
Aug 13 00:59:03 raspberrypi aqualinkd[754]: RS Serial: Port /dev/ttyUSB0 low latency mode is set
Aug 13 00:59:03 raspberrypi aqualinkd[754]: AqualinkD: Listening to Aqualink RS8 on serial port: /dev/ttyUSB0
Aug 13 00:59:03 raspberrypi aqualinkd[754]: AqualinkD: Waiting for Control Panel probe
Aug 13 00:59:03 raspberrypi aqualinkd[754]: AqualinkD: Got probe on '0x09' Standard Protocol
Aug 13 00:59:03 raspberrypi aqualinkd[754]: AqualinkD: Starting communication with Control Panel
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 00:59:26 raspberrypi aqualinkd[754]: Panl&Prog: Starting programming thread 'Get Heater setpoints'
Aug 13 00:59:26 raspberrypi aqualinkd[754]: Panl&Prog: Starting programming thread 'Get Freeze protect'
Aug 13 00:59:26 raspberrypi aqualinkd[754]: RS Serial: Serial read bad Jandy checksum, ignoring
Aug 13 00:59:26 raspberrypi aqualinkd[754]: RS Serial: Read Jandy packet BAD PACKET To 0x00 of type PDA HlightChars | HEX: 0x10|0x02|0x00|0x10|0x02|0x08|0x02|0x00|0x00|0x00|0x00|0x00|0x1c|0x10|0x03|
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 00:59:26 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 00:59:27 raspberrypi aqualinkd[754]: Panl&Prog: Did not receive expected reply from last RS SEND command, resending '0x09'
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:00:12 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:27 raspberrypi aqualinkd[754]: Panl&Prog: Thread (Get Heater setpoints) 0x8b1c68 timeout waiting for thread (Get Freeze protect) 0x8b1650 to finish
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:01:52 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:01:54 raspberrypi aqualinkd[754]: Panl&Prog: Did not receive expected reply from last RS SEND command, resending '0x09'
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:02 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:07 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:07 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:07 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:07 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:08 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:08 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: Panl&Prog: Did not receive expected reply from last RS SEND command, resending '0x09'
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel version 6521 REV E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Control Panel revision E
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Could not select REVIEW menu
Aug 13 01:02:10 raspberrypi aqualinkd[754]: AllButton: Can't get freeze setpoints from Control Panel

None of the pool buttons on the web interface responds correctly, usually I test the lights and after I turn them on, they turn off on their own a couple of seconds later.

I have double-checked the wiring (+ and -) and it is correct. I am using the FTDI USB-RS485-WE-180 serial 1.8M wire end cable, does this looks like a bad Serial cable?

Thanks

[sfeakes]

This looks like a bad serial connection, the polarity is correct, but the quality looks to be the issue. So maybe try different cable / connecting at a different point etc.

[augalos]

Thanks a lot for your quick response!

I wonder if it is the USB type A to USB micro C adapter that is used to connect the FTDI cable to the Pi Zero, or it is the FTDI cable itself? Any recommendations for an adapter that will be more reliable? For the cable I bought the one you recommended from Amazon.

[sfeakes]

Id be more inclined to say it’s not the USB cables, but that physical RS485 cabling. Ie cable (or connection) from panel to USB2RS485 adapter.

[augalos]

Thanks, for the feedback.

I did an experiment and replaced the Pi Zero with a Pi 4 I had lying around; when I did that, I didn't use the adapter anymore, and that eliminated all the RS Serial errors. It seems to be stable now, so definitely the adapter introduced some noise in the line. I still get the Panl&Prog errors. Are these also related to cabling or they are due to mistakes on the aqualinkd conf file? Most of those seem to be present after boot and they don't appear anymore after a while. Making progress :)

[sfeakes]

Programming errors come from AqualinkD not getting messages it expects from the panel. Checksum / serial errors are usually the cause of that. I can't recall if Rev E has been tested, so it might be something to do with that revision. If you turn on debugging logging (at startup using the config file) and post the output, that should help figure it out.

[augalos]

aqualinkd_log.txt

Thanks, I enabled DEBUG and recaptured the output when the system boots. I have attached the log file

[sfeakes]

AqualinkD is constantly loosing connection to the panel. The question is Why? I’m not sure from the logs.
Try slowing down AqualinkD on how fast it reply’s to messages, set rs485_frame_delay = 20 in aqualinkd.conf, currently it’s set to 4.

If that doesn’t make a difference, can you post debug output of serial_logger, running below (without AqualinkD running)
sudo ./serial_logger /dev/ttyUSB0 -p 1000 -d

[augalos]

When I Switched to frame delay 20 AqualinkD couldn't connect to the panel. I have included that debug log together with the serial logger log
frame_delay_20.log.txt
serial_logger.log.txt

Thanks

[sfeakes]

I think we are getting to the bottom of this. As you saw Slowing down AqualinkD, made the loosing connection with control panel problem worse, ie the panel wasn’t receiving messages quickly enough so thinks the connection is dead. But the value of 20 is 1/2 the actual time, a value above 40ms should cause that issue. (The timing debug messages in AqualinkD shows you the time it is actually taking). Is AqualinkD is showing replying to messages in 0.020, but my guess is it’s probably above 0.040 at the control panel. Other than AqualinkD, there are two things in the way of the speed in which messages are sent, one is the Linux kernel, the 2nd is the USB2RS485 device. So first set rs485_frame_delay to 0, (make AqualinkD send messages as quickly as possible. This won’t fix the problem, but it will help.
Second need to figure out where the delay is. Make sure nothing else is running on the Pi, stop any services you can, monitor the system log messages for anything that may indicate some problems. The FTDI adapter you have, should not cause you any issue, but their are a lot of fake ones on the market, maybe double check that’s a real FTDI adapter, and also check that you have only connected the two data wires, all the other wires on the adapter should be open / not connected / and not touching anything. Other than that, try to connect the adapter directly to the control panel, see if that helps.

One other thing try uncommon this setting in aqualinkd.conf ftdi_low_latency = no. The default is yes, but on some bad USB2RS485 connectors that can cause an issue, so setting it to no may help. You should probably reboot the system once you change that.

[augalos]

OK, I tried different things and I believe we have arrived (for now). I don't think it is 100% fixed but everything seems to work now, both through the AqualinkD web interface and Homekit as well. Here is what I did:

1. Set the rs485_frame_delay to zero - I think this made the biggest difference
2. Reinstalled Rasbian Lite, got rid of all unnecessary processes etc.
3. Used Electrical tape and taped all unused cables together
4. I tried to connect the ground but I don't think this made any difference, it may have made things worse in terms of errors.
5. I tried setting the ftdi_low_latency = no but I don't think it made a difference, I think my cable is genuine.

See the attached logs
rs485_frame_delay_zero_fresh_linux.log.txt
rs485_frame_delay_zero_and connected ground.log.txt
rs485_frame_delay_zero.log.txt

Thanks for all the help!