Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FoxESS has Cloud timeout fetching Device Detail #268

Closed
NunoOliveira opened this issue Sep 19, 2024 · 14 comments
Closed

FoxESS has Cloud timeout fetching Device Detail #268

NunoOliveira opened this issue Sep 19, 2024 · 14 comments
Labels
bug Something isn't working

Comments

@NunoOliveira
Copy link

Describe the bug
The integration just stopped working 7 days ago and not able to pull any data.
Is was working fine for several months before and no changes were made with the exception of HA version updates.

Debug logs:

2024-09-19 10:39:28.019 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.

HA version:
core_2024.9.2

foxess-ha version:
0.42

@NunoOliveira NunoOliveira added the bug Something isn't working label Sep 19, 2024
@FozzieUK
Copy link
Collaborator

Have you changed your apikey recently ?

If you can restart home assistant, wait 60 seconds when it has started up and then got to Settings, System, Logs and click the ‘Load Full Logs’ button - if you could copy and paste every log entry that relates to foxess here.

@NunoOliveira
Copy link
Author

NunoOliveira commented Sep 19, 2024

I did change the api key this morning as an attempt to fix the issue. I generated a new key a replaced it on the configuration.yaml.
but it had already stopped working 7 days ago without any change.

These are the latest Logs:

2024-09-19 15:25:37.315 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:25:42.321 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:26:42.947 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:26:47.948 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:27:47.990 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:27:52.992 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:28:53.243 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:28:58.246 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:29:58.011 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:30:03.028 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:31:03.113 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:31:08.114 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:32:08.033 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:32:13.040 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:33:13.128 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:33:18.130 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:34:17.976 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:34:22.978 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:35:23.149 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:35:28.151 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:36:27.893 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:36:32.906 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:37:33.116 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:37:38.121 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:38:37.908 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:38:42.915 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:39:43.795 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:39:48.797 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:40:49.375 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:40:54.376 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:41:53.888 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:41:58.895 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:42:59.972 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:43:04.976 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:44:04.996 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:44:09.998 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:45:10.142 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:45:15.143 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:46:15.169 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:46:20.171 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:47:19.974 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:47:25.005 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:48:24.920 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:48:29.921 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:49:29.975 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:49:34.981 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:50:35.315 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:50:40.316 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:51:39.949 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:51:44.951 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:52:44.903 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:52:49.914 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:53:50.030 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:53:55.032 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:54:55.377 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:55:00.378 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:56:00.272 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:56:05.278 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:57:04.941 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:57:10.037 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:58:10.060 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:58:15.061 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 15:59:14.996 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 15:59:19.997 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.
2024-09-19 16:00:20.150 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 16:00:25.159 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.

2024-09-19 16:01:24.910 ERROR (MainThread) [custom_components.foxess.sensor] OA Device Detail Bad Response: {'errno': 40402, 'msg': 'Your request exceeds the limit. Please try again later', 'result': None}
2024-09-19 16:01:29.916 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout fetching Device Detail, will retry in 1 minute.

@FozzieUK
Copy link
Collaborator

If you log into the Fox cloud, and go to the user profile, api management screen (as if you were going to request a new apikey) what are the values for your ‘Number of Calls per day’ and ‘Remaining calls for today’

@NunoOliveira
Copy link
Author

I did an HA restart this morning, I let it run for 1 hour and still no readings.

These are the logs since restart:

2024-09-20 09:47:13.220 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration foxess which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant 2024-09-20 09:47:17.055 WARNING (ImportExecutor_0) [homeassistant.const] POWER_VOLT_AMPERE_REACTIVE was used from foxess, this is a deprecated constant which will be removed in HA Core 2025.9. Use UnitOfReactivePower.VOLT_AMPERE_REACTIVE instead, please report it to the author of the 'foxess' custom integration 2024-09-20 09:47:27.142 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform foxess is taking over 10 seconds.

API calls:
Number of calls per day:1440
Remaining calls for today:580

Before the restart I had over 600 calls left.

FYI the Inverter Datalogger is connected and working fine I can see the power reading on Foxcloud.

Foxcloudpower

@FozzieUK
Copy link
Collaborator

I’m going to need some debugging enabled to see whats happening, your api-call count is low if it is only this integration making calls.

Can you add this to your configuration.yaml near the top

logger:
  default: warning
  logs:
    custom_components.foxess: debug 

Then restart your home assistant again, leave it a couple of minutes and copy the logs again - there will be a lot more detailed information and you might want to obscure your apikey and inverter serial number as they will be shown.

@NunoOliveira
Copy link
Author

Here's the debug logs on the attached file.

Looks like there's a lot api calls being sent, today's remaining api calls are about to be exhausted only have 6 left.

debug HA Foxess.txt

@FozzieUK
Copy link
Collaborator

Thanks, that's very strange - I can see 2 things, the first is that your inverter is reporting it is in alarm state, I can't see what the alarm state is because the get real time variables is failing with a response of 'query data failed' - i'm not sure why it would do that, it could be related to the inverter alarm.

Because it fails to startup properly, it's skipping round this loop every minute and burning through 3 api calls (that will need fixing).

But for now we need to understand why the inverter is in alarm - can you check by logging into the website, what alarm is your inverter reporting ?

@FozzieUK
Copy link
Collaborator

I have released a pre-release version 0.42-beta-3 which fixes the integration exhausting your api count when an inverter is in alarm.

If you could upgrade to that as soon as possible, and even though your inverter is reporting a fault it won’t waste your api_calls - just need to understand what the fault is now.

@NunoOliveira
Copy link
Author

NunoOliveira commented Sep 20, 2024

The fault is related to not having a battery connected to the inverter.
I never had a battery since the install of the inverter so it shouldn't be related.

fault Foxess

I've also installed version 0.42-beta-3.
I uploaded the files manually to HA but is there any way you can install the beta versions directly from HACS? Couldn't find way of doing it.

@FozzieUK
Copy link
Collaborator

If you are on the old HACS, go to the integration in HACS, click the 3 dots in the top right corner and click 'Redownload' - it will show a dialogue box with the current version and above it is a switch for 'Show beta versions' - turn that on and it will show it.
I think the new version of HACS (2) always shows pre-releases by default.

You are on a very old version of firmware, if you upgrade there is a switch within your inverter Settings, Feature 'BMS Set' that allows you to disable the no battery alarm - it's also possible the age of your firmware versions is causing a problem.

Fox ESS did an update on the OpenAPI on 13th September and it's broken several Energy Stats app users (also uses OpenAPI) in the same way with an error 41200 'query data failed' - but this is the first one I have seen using this cloud integration.

As it is an internal error to the API, you will need to ask Fox to investigate and fix the problem - I would suggest you email your local Fox service and tell them that for your inverter serial number the OpenAPI is failing with this error {'errno': 41200, 'msg': 'Query data failed', 'result': None} on an OpenAPI call to endpoint /op/v0/device/real/query to get device variables with request {"sn":"65BH*********10" } (but fill in your inverter serial number for them to see)

It could be your lack of battery, or old firmware versions - but obviously it used to work and it failed after Fox updated the API recently.

If you are willing to email me your apikey and inverter serial number to b5djf@yahoo.com (you can change your apikey later), I will do some testing and try and find out which variable within the call is failing - it may be I can get something working for you whilst you wait for Fox to investigate the problem.

@FozzieUK
Copy link
Collaborator

Hi @NunoOliveira,

Thanks for sending over your apikey and inverter SN.

I've been running debug on your system and found that the corruption is in a certain field returned by the get device variables call - the variable in question is currentFault this is supposed to contain a text string relating to the error your system has generated - in your case it is causing the data query to crash as there is invalid data in it.

You might find that updating to the latest version of firmware on your inverter clears this problem - if you do that you can go into your inverter settings and tell it you don't have a BMS installed which will make the error clear anyway.

But for now, i've created a pre-release v0.42-beta-4 which will avoid addressing that particular variable and your system will work.

If you update to this release, and then add the Restrict: true switch to your configuration.yaml file (see below) then restart your HA and you should be back to normal.

If you update your inverter, you can test if it works by removing the 'restrict' switch and it will revert to its normal 'default' operation i.e. get all available variables.

sensor:
  - platform: foxess
    deviceID: enter_your_inverter_id
    deviceSN: enter_your_inverter_serial_number
    apiKey: enter_your_personal_api_key
    Restrict: true

@NunoOliveira
Copy link
Author

NunoOliveira commented Sep 23, 2024

Hi @FozzieUK,

Thanks for the help solving the issue.

I've installed the new v0.42-beta-4 and it seems to be working fine now
Fox Energy

I'll try to get my inverter updated to a newer firmware version and I'll let you know what the outcome is.

Cheers

@FozzieUK
Copy link
Collaborator

Ok glad you have it working again

Please let me know how you get on after the inverter is updated - I suspect that when Fox did the update they have changed the data type of the current fault variable and as you had an active error it had the old string text in it which is causing them an internal server side error.

Hopefully once you've been able to clear the 'old' text error it will work normally again 👍

@FozzieUK
Copy link
Collaborator

now resolved with release v0.43

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants