Datalogger crashes with IntegrityError because 'timestamp' is null

[danielterhorst]

Maybe related to the fact that it's a DSMR 4.2 unit (#73). Stacktrace at:

• https://sentry.dth.wtf/share/issue/322e38/
• https://sentry.dth.wtf/share/issue/322e39/

Query that's executed:

'INSERT INTO "dsmr_datalogger_dsmrreading" ("timestamp", "electricity_delivered_1", "electricity_returned_1", "electricity_delivered_2", "electricity_returned_2", "electricity_currently_delivered", "electricity_currently_returned", "extra_device_timestamp", "extra_device_delivered", "processed") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "dsmr_datalogger_dsmrreading"."id"'
with parameters:

[None, None, None, None, None, None, None, None, None, False]

[dennissiemensma]

I can't explain why this is happening. Might be a bug in the date parserl but it seems all other fields of the INSERT are None as well. So it might relate to #73 as you mentioned yourself.

Can you tell me whether it's happing again after you've updated your local installation with the fix for #73?

[danielterhorst]

I think I tracked this one down. It seems so happen when starting the datalogger and an incomplete telegram is processed. For example, for one of the cases where it error'd the telegram was:

1-0:99.97.0(1)(0-0:96.7.19)(000101000001W)(2147483647*s)
1-0:32.32.0(00000)
1-0:32.36.0(00000)
0-0:96.13.1()
0-0:96.13.0()
1-0:31.7.0(001*A)
1-0:21.7.0(00.213*kW)
1-0:22.7.0(00.000*kW)
!DEA1

According to the protocol, every telegram starts with:

/ X X X 5 Identification [...]

I guess this could be used together with the termination to determine if the telegram is complete.

[dennissiemensma]

Thanks for investigating. The current telegram parser does indeed not check the start of the telegram, as it asumes it's complete. It only checks the ending delimiter:

    while True:
        data = serial_handle.readline()

        ...

        buffer += data

        # Telegrams start with '/' and ends with '!'. So we will use them as delimiters.
        if data.startswith('!'):
            return buffer

I could add another check for data.startswith('/'), which will reset the buffer, skipping the incomplete reading, starting with a fresh reading, which should be complete.

[dennissiemensma]

Added test to verify and fixed it. Tests passed as well, so merged to master.

[dennissiemensma]

This should fix your problem.

Thanks for reporting it as I have not seen it happening here myself, admitting that I lack Sentry monitoring for the moment. Please let me know if it happens again after applying the latest fix.