Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1654

Closed
fjbosscha opened this issue Jul 4, 2022 · 12 comments
Closed

🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1654

fjbosscha opened this issue Jul 4, 2022 · 12 comments

Comments

@fjbosscha
Copy link

Description

Sinds afgelopen week ben ik overgestapt op DSMR-reader. Omdat ik een aantal jaren historie in domoticz heb te zitten, heb ik deze geïmporteerd binnen DSMR-reader. Ik zie dat data wel in de betreffende tabellen is gekomen (dsmr_consumption_electricityconsumption en dsmr_consumption_gasconsumption) maar via de archief optie, zie ik alleen maar gas-waarden.

Ik heb al in de andere issues gekeken, maar zie niet direct wat waar mis gaat.

DSMR-reader version

5.3

DSMR-reader platform

No response

Debug info dump

DSMR-READER
    App / Python / Database                                                        v5.3 / v3.9.2 / postgresql
    BE sleep / DL sleep / Retention / Override                                     1.0s / 1.0s / 672h / False
    Latest telegram version read / Parser settings                                                 "50" / "4"

DATA
    Telegrams total (est.)                                                                             158839
    Consumption records electricity / gas (est.)                                                103613 / 3230
@dennissiemensma
Copy link
Member

Je zou #1630 kunnen bekijken. Daar staat een soortgelijk issue.

#1630 (comment)

@dennissiemensma dennissiemensma added this to the Other milestone Jul 4, 2022
@fjbosscha
Copy link
Author

fjbosscha commented Jul 4, 2022 •
edited by dennissiemensma
Loading

Bedankt voor je reactie. Die had ik inderdaad ook al gezien, maar mijn data is, in mijn optiek, correct.

dsmrreader=> select read_at from dsmr_consumption_electricityconsumption;
        read_at         
------------------------
 2019-02-08 00:00:00+01
 2019-02-08 23:59:00+01
 2019-02-09 00:00:00+01
 2019-02-11 23:59:00+01
 2019-02-12 00:00:00+01
 2019-02-14 23:59:00+01
 2019-02-15 00:00:00+01
 2019-02-15 23:59:00+01
 2019-02-16 00:00:00+01
 2019-02-16 23:59:00+01
 2019-02-17 00:00:00+01
 2019-02-17 23:59:00+01
 2019-02-18 00:00:00+01
 2019-02-18 23:59:00+01
 2019-02-19 00:00:00+01
 2019-02-19 23:59:00+01
 2019-02-20 00:00:00+01
 2019-02-20 23:59:00+01
 2019-02-21 00:00:00+01
 2019-02-21 23:59:00+01
 2019-02-22 00:00:00+01
 2019-02-22 23:59:00+01
 2019-02-23 00:00:00+01
 2019-02-23 23:59:00+01
 2019-02-24 00:00:00+01
 2019-02-24 23:59:00+01
 2019-02-25 00:00:00+01
 2019-02-25 23:59:00+01
 2019-02-26 00:00:00+01
 2019-02-26 23:59:00+01
 2019-02-27 00:00:00+01
 2019-02-27 23:59:00+01
 2019-02-28 00:00:00+01

dsmrreader=> select read_at from dsmr_consumption_gasconsumption;
        read_at         
------------------------
 2022-06-28 21:45:08+02
 2022-06-28 21:50:01+02
 2020-01-10 23:59:00+01
 2020-01-11 00:00:00+01
 2020-01-11 23:59:00+01
 2020-01-12 00:00:00+01
 2020-01-12 23:59:00+01
 2020-01-13 00:00:00+01
 2020-01-13 23:59:00+01
 2020-01-14 00:00:00+01
 2020-01-14 23:59:00+01
 2020-01-15 00:00:00+01
 2020-01-15 23:59:00+01
 2020-01-16 00:00:00+01
 2020-01-16 23:59:00+01
 2020-01-17 00:00:00+01
 2020-01-17 23:59:00+01
 2020-01-18 00:00:00+01
 2020-01-18 23:59:00+01
 2020-01-19 00:00:00+01
 2020-01-19 23:59:00+01
 2020-01-20 00:00:00+01
 2020-01-20 23:59:00+01
 2020-01-21 00:00:00+01
 2020-01-21 23:59:00+01
 2020-01-22 00:00:00+01
 2020-01-22 23:59:00+01
 2020-01-23 00:00:00+01
 2020-01-23 23:59:00+01

Heb truncate dsmr_stats_daystatistics gedraaid en de processen weer aangezet.

Debug aangezet en zie nu in de dsmr_backend.log allemaal de volgende meldingen:

2022-07-04 19:26:23,478 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:23,497 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:24.490464+02:00 (ETA 0:00:00.999936)
2022-07-04 19:26:23,509 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:23,513 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:24,637 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:24,637 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:24,651 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:25.647331+02:00 (ETA 0:00:00.999938)
2022-07-04 19:26:24,660 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:24,664 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:25,670 DEBUG    schedule     execute_scheduled_processes      32 | SP: 3 backend service(s) ready to run
2022-07-04 19:26:25,671 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:25,760 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:25,761 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:25,778 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:26.772076+02:00 (ETA 0:00:00.999945)
2022-07-04 19:26:25,778 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:26:25,783 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:26:00+02:00 as fuzzy start
2022-07-04 19:26:25,787 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:15:00+02:00, postponing for a bit...
2022-07-04 19:26:25,793 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:26:30+02:00 (ETA 0:00:04.212378)
2022-07-04 19:26:25,803 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:25,809 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:26,816 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:26,817 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:26,916 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)

Weet niet of dit moet kloppen!

@fjbosscha
Copy link
Author

fjbosscha commented Jul 4, 2022 •
edited by dennissiemensma
Loading

Zat wat vervuiling in (had de database op een test-systeem gezet, waarbij de mqtt en pvoutput ook liepen.

logging is nu:

2022-07-04 19:32:52,267 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:52,292 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:53.283032+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:52,316 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:53,349 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:54.341110+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:53,368 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:54,376 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:54,377 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:54,400 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:55.392694+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:54,421 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:55,454 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:56.446108+02:00 (ETA 0:00:00.999913)
2022-07-04 19:32:55,454 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:32:55,461 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:32:00+02:00 as fuzzy start
2022-07-04 19:32:55,468 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:30:00+02:00, postponing for a bit...
2022-07-04 19:32:55,478 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:33:00+02:00 (ETA 0:00:04.530197)
2022-07-04 19:32:55,499 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:56,508 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:56,510 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:56,534 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:57.526560+02:00 (ETA 0:00:00.999912)
2022-07-04 19:32:56,556 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:57,565 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:57,567 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:57,592 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:58.583766+02:00 (ETA 0:00:00.999911)
2022-07-04 19:32:57,609 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:58,618 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:58,620 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:58,647 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:59.638447+02:00 (ETA 0:00:00.999905)
2022-07-04 19:32:58,673 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:59,682 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run

@dennissiemensma
Copy link
Member

Hoe draai je DSMR-reader? Docker of native?

@dennissiemensma
Copy link
Member

Staat er wel data in de dsmr_stats_daystatistics tabel?

@fjbosscha
Copy link
Author

fjbosscha commented Jul 4, 2022 •
edited by dennissiemensma
Loading

Ik draai deze native.
Er staat geen data in dsmr_stats_daystatistics

dsmrreader=> select  * from dsmr_stats_daystatistics;                                                                                                                  id | day | total_cost | electricity1 | electricity2 | electricity1_returned | electricity2_returned | electricity1_cost | electricity2_cost | gas | gas_cost | average_temperature | highest_temperature | lowest_temperature | fixed_cost | electricity1_reading | electricity1_returned_reading | electricity2_reading | electricity2_returned_reading | gas_reading 
----+-----+------------+--------------+--------------+-----------------------+-----------------------+-------------------+-------------------+-----+----------+---------------------+---------------------+--------------------+------------+----------------------+-------------------------------+----------------------+-------------------------------+-------------
(0 rows)

@fjbosscha
Copy link
Author

ik heb nog even in de "running" tabellen gekeken en daar staan in dsmr_stats_daystatistics wel waarden bij Electricity1_reading, electricity1_returned_reading, electricity2_reading, electricity2_returned_reading en gas.
Wat mij hierbij opvalt is dat hier geen waarden zijn bij electricity1, electricity2, electricity1_returned en electricity2_returned. Bij gas staan er wel de dagelijkse waarden.

@fjbosscha
Copy link
Author

fjbosscha commented Jul 4, 2022 •
edited by dennissiemensma
Loading 

  329 | 2020-01-08 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9648.898 |                      2956.777 |             8158.299 |                      6917.227 |            
  330 | 2020-01-09 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9651.773 |                      2956.777 |             8164.627 |                      6917.659 |            
  331 | 2020-01-11 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.304 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               14.423 |                         0.120 |                0.000 |                         0.000 |       9.134
  332 | 2020-01-12 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.424 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               20.704 |                         0.615 |                0.000 |                         0.000 |      15.438
  333 | 2020-01-13 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.257 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               29.230 |                         0.615 |                0.000 |                         0.000 |      22.862
  334 | 2020-01-14 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.747 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               30.535 |                         0.615 |                5.621 |                         0.252 |      30.119
  335 | 2020-01-15 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  5.822 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               32.128 |                         0.615 |               11.657 |                         1.695 |      36.866
  336 | 2020-01-16 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.104 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               35.107 |                         0.615 |               18.425 |                         1.695 |      42.688
  337 | 2020-01-17 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.144 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               36.625 |                         0.615 |               25.185 |                         2.612 |

Er is, op 11 januari 2020 wel een meterwissel geweest. Kan dat invloed hierop hebben. Vanaf 11 januari 2020 is de meter aangepast, zodat ook gas nu met de slimmemeter mee komt

@dennissiemensma
Copy link
Member

Klopt je data in dsmr_consumption_electricityconsumption dan wel? Dat is de tussentabel voordat het naar dsmr_stats_daystatisticsgaat

@dennissiemensma
Copy link
Member

dennissiemensma commented Jul 4, 2022 •
edited
Loading

Je kunt nog dsmr_stats_daystatistics en dsmr_stats_hourstatistics leeggooien en dan in de URL /admin/dsmr_backend/scheduledprocess/ het proces Generate day and hour statistics zoeken, aanpassen naar vandaag + nu.

Dan hergenereert die als het goed is weer de statistieken. In de debug logs zie je dat dan ook.

Je kunt ook nog wat andere proberen, zoals:

sudo su - dsmr
./manage.py dsmr_stats_reconstruct_missing_day_statistics

Nu ik het weer teruglees, die was hier ook voor gemaakt. Zie #1302 (comment). Probeer die anders eerst maar.

@PGTrynwalden
Copy link

Hoi Dennis,
Heb het laatste commando gedraaid en de ontbrekende data staat erin.

Dank voor het meedenken.

@dennissiemensma
Copy link
Member

Fijn om te horen! Ik zal het bij de andere issues zetten.

@dsmrreader dsmrreader locked and limited conversation to collaborators Jul 5, 2022
@dennissiemensma dennissiemensma converted this issue into discussion #1657 Jul 5, 2022

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dennissiemensma @fjbosscha @PGTrynwalden