Belchertown taking forever on some weewx v5 systems

[vinceskahan]

We're seeing multiple folks with issues running Belchertown in v5 weewx when the same systems worked fine on v4. The underlying issue is a feature addition in v5 that attempts to calculate derived elements on the fly as StdReport runs.

The long thread is at https://groups.google.com/g/weewx-user/c/yfqGGuFw-L8/m/JTAvIFtUAAAJ with this link on the reply documenting what I found re: the schema after a lot of back and forth. The thread itself top-to-bottom is likely worth going through please, as it has some typical timing data for a not-broken scenario for a pi3+ just as a reference.

What's happening - the patterns seems to be centered on folks with old db running the old original small wview-compatible database. In those cases none of the items in your skin.conf Extended Observations label section exist in the (small schema) db, so weewx v5 tries to calculate on the fly.

This is glacially slow. With many years of data this basically never completes, weewx eventually has a locked db and the StdReport thread aborts and weewx tries to self-heal via restart. There is no way to stop this nor recover other than disabling Belchertown. I tried on a pi3+ and failed to complete one StdReport run in 600 seconds before it aborted. Typical on that kind of hardware should be under 10 seconds for a db double the size of the test db we used.

How to fix the problem from the user's perspective - The 'fix' is to have the user use the wview_extended db so these elements are present. They'd also have to use weectl database --calc-missing to generate any+all the missing summary tables. This takes hours potentially on a slow system.

Suggested actions:

• tell people to use the wview_extended schema in your README file and wiki/docs as a prerequisite
• if possible check and syslog if your skin.conf references expected database elements that don't exist in the db schema

Possible other related issue:
I also notice that not all the skin.conf extended labels exist even in the wview_extended schema, but I can't follow your code well enough to understand why they're defined in skin.conf at all. I couldn't even find some of them referenced in .tmpl files but perhaps in the js code I can't grok.

[poblabs]

Thanks for the update @vinceskahan! I haven't even had a chance myself to try out weewx 5.0. Life has come at me fast and hard and projects have had to take a back seat. I'm with you, I'd have to take some time to understand the code as well since it's been a while :) I also got an email from @tkeffer about this too.

At this point, is the resolution just on the user? Or is there more work to be done in the skin.conf?

I agree, the skin could probably use an update - even highcharts is getting old on it - but where to find the time?

[vinceskahan]

Life happens. Kids have us outnumbered. Been there. Oh man were we in your shoes a number of years ago. Fun though.

(let me know via email if you need a hand trying v5 - it is pretty great)

Re: this issue - I think just a note in the README and/or wiki for v5 users specifically that documents the need to either use the extended schema 'or' add whatever label(s) are being referenced in the skin and are missing from their db should be good enough.

Symptoms are the Belchertown skin never finishes running and StdReport aborts, likely with a db locked message. When they go back to v4 the problem goes away because it's v5's new behavior triggering all the compute that's slowing things down.

How to switch to the new db was pretty easy to do in v5 and I'd probably recommend that FWIW, as it would fix any skins looking for the wview_extended schema elements including Belchertown.

# timings are on a m1 mac mini - any pi would take many minutes but it does work.
# Faster hardware helps a lot.
#
# these assume you are calling wview_extended in weewx.conf which is what you'd see for a new v4 or v5 user
# if not, they'd need to edit their file to specify the extended schema

stop weewx
(save your original weewx.sdb someplace just.in.case.)

weectl station reconfigure
(answer y - takes 35 secs to do this step)

mv weewx.sdb weewx.sdb.keepme
mv weewx.sdb_new weewx.sdb

weectl database rebuild-daily
(answer y - takes 88 secs to do this step)

start weewx up again

Adding a Utilities Guide pointer should be plenty good enough. The pointer currently for that is https://www.weewx.com/docs/5.0/utilities/weectl-database/#reconfigure-a-database

[mmc1]

I upgraded from weewx 4.10.2 to 5.0 with Belchertown Skin Version: 1.3.1. The live data, mqtt broker, and hicharts plots all appear to be working properly, but the Forecast module is not updating (stuck on last updated time before I upgraded weewx). Weather Records Snapshots and NOAA reports are also not updating, and the About page does not show the latest WeeWX version. I suspect the Recent Local Earthquake module is also not updating, but this is not confirmed.

I tried the station reconfigure/database rebuild-daily steps outlined above, and while those steps appear to have executed properly, they do not resolve this issue. I'll keep searching for hints online.

[vinceskahan]

As always, set debug=1, reset weewx, and look at your logs. Almost certainly something is aborting mid-run.

[ChrisM510]

I had nothing but problems when I upgraded to 5.0. I ended up restoring from backup and going back to 4.10.

[jamie75]

I’ll add. I upgraded to 5.0 last week and have zero issues. But my setup is pretty basic

[alexdelprete]

weectl station reconfigure

I just completed the upgrade to v5. Used debian install. I have 3 stations, using weewx@ multi configuration. Everything seems to work fine, also for Belchertown: https://www.iconoclasti.com/meteo/

May I ask you what the weectl station reconfigure command does? I didn't use it. I simply installed weewx 5, changed config files to reflect new paths, moved data from old paths to new paths, and that's it. Didn't use weectl at all for now. I was already on extended schema.

[vinceskahan]

You do not need to do anything with reconfigure if you started with v4 or v5 weewx. Those versions already use the larger extended db schema. The problem appears only for users who started with weewx v3 or earlier.

[bonjour81]

Hello
I assume maybe @vinceskahan meant
weectl database reconfigure

Instead of
weectl station reconfigure

as station reconfigure won't create new db.

I also assume in 1st port that it is meant
weectl database calc-missing
instead of
weectl database --calc-missing

it may need sudo for some.
not a big deal, just to mention in case it helps some people

[george-crum]

Wanted to Thank everyone here for the fix. My graphs were spotty for showing day trends but if you hovered over them the data was there even though nothing showed. These are the steps I did to fix the issue. I found that I had to run rebuild-daily before calc-missing due to an error I cannot remember and my terminal history is gone for that.
stop weewx backup db weectl database rebuild-daily weectl database calc-missing start weewx

[roe-dl]

May be, there is a connection to that issue: weewx/weewx#939

[roe-dl]

There is a new WeeWX wiki article that could be related: XTypes performance

[cdbailey]

Thank you @vinceskahan for your instructions. The WeeWX Python process was being terminated after consuming 6GB+ of RAM due to the Linux kernel triggering the OOM killer. Upon investigation initiated by your post, I identified that this install was not using the wview_extended schema. This install originated in circa 2016, so it most definitely was v3 or prior. The database contained data as far back as 2012 (converted from Meteobridge), approximately 4.2 million records.

After adjusting my config to use wview_extended, issuing the weectl database reconfigure command, redirecting weewx.conf to the new database, and running the weectl database rebuild-daily command, everything appears to be working great!

[ggowins]

I think my problem might be related to this. I upgraded from 4.10.2 to 5.0.2, and afterwards, I saw in the logs that once it started the Belchertown calcs (cheetah, etc), python spikes to 100% and like cdbailey above eventually weewx will crash. I uninstalled the Belchertown extension, and now the CPU for python stays low.

So after reading the posts above, I wonder if running the database reconfigure, rebuild-daily, and calc-missing will fix my issues. I do use some schema extensions to store lightning data, so not sure what might happen there.

I do have a little more than 6 years worth of data in the DB, and I'm using mariadb for the DB

[vinceskahan]

@ggowins at this point this is not a Belchertown issue it is a 'is your weewx setup correctly' issue. Short description is that if you are using elements not in your db schema then you will experience this with 'every' skin referencing that missing element. If you are using the original v3 small db then the reconfigure to the new v4 bigger schema should fix the problem. If you are using something custom then you need to add whatever column(s) might be missing in your schema. I have no idea how to do so with mariadb, but the howto there isn't anything you should be asking here regardless, as it has nothing to do with Belchertown. I'm sure the nice folks at weewx-user would answer any questions you might have.

[tkeffer]

See the wiki article Performance of xtypes for an explanation and for various options for fixing the problem.

[ggowins]

@ggowins at this point this is not a Belchertown issue it is a 'is your weewx setup correctly' issue. Short description is that if you are using elements not in your db schema then you will experience this with 'every' skin referencing that missing element. If you are using the original v3 small db then the reconfigure to the new v4 bigger schema should fix the problem. If you are using something custom then you need to add whatever column(s) might be missing in your schema. I have no idea how to do so with mariadb, but the howto there isn't anything you should be asking here regardless, as it has nothing to do with Belchertown. I'm sure the nice folks at weewx-user would answer any questions you might have.

Thanks. for the info. I am running v4 schema, as reported by weectl database check. Searching the weewx-user list I didn't see a similar issue to the one I was having, and since it went away when I uninstalled Belchertown, that's what led me here. Now I realize I was mistaken. I'll take a look at the link Tom posted and use that to bird dog my issue. Thanks!

[tkeffer]

From experience, it seems that the type appTemp usually causes the problem. A solution may be as simple as

weectl database add-column appTemp
weectl database drop-daily
weectl database rebuild-daily

See the documentation for weectl database add-column.

Be sure to do a backup first!

[ggowins]

From experience, it seems that the type appTemp usually causes the problem. A solution may be as simple as

weectl database add-column appTemp
weectl database drop-daily
weectl database rebuild-daily

See the documentation for weectl database add-column.

Be sure to do a backup first!

Thanks for the pointer! I looked in the skin.conf, and added the other columns that were listed with appTemp in the Extra Observation labels section and weren't already in the schema, and rebuilt the dailies. That has helped tremendously. I still get a CPU spike when the weewx.reportengine section runs, but only for 10 to 15 seconds. It generates 12 files for Belchertown in 5 seconds according to the log.

I'll keep digging to see what other fields needs to be added. But this is working much better, and shouldn't crash after several hours.

[WiggyToo]

Just to add to this, if like me, you are upgrading from an old weewx version (3) that has the simple schema in weewx.cnf then you need to change that to extended before doing the reconfigure.

I had upgraded, then put my original old weewx.cnf back as it had my settings in it.

This meant the reconfigure didn't update the database schema.

I only noticed when I tried to add noise data to the database and that field was missing.

[tkeffer]

You do not need to change the schema. All you need to do is add any missing derived types.
For the Belchertown skin, that means adding just one type: appTemp.

weectl database add-column appTemp
weectl database calc-missing

Far simpler!

[WiggyToo]

@tkeffer thanks Tom, in my case I wanted to change the schema for other reasons, but I see what you mean about Belchertown specifically.