Protocol in SQLite DB

[rossandrews]

With the protocol definitions currently being in large nested dictionaries it makes changing them very time consuming. It can also lead to inconsistency since each protocol can be structured differently then the code needs to be modified to account for that. Since they all need to share a similar structure, moving them into a relational DB will be relatively easy and make future changes much easier. We can also make use of Pythons very mature data frameworks to help with validation.

How this will work is that we will have a DB file with the protocols that will be checked into git. When Powermon runs it will load the protocols from the DB instead of the current dictionaries.

There are a few drawbacks to moving to a SQLite db, here's my current list and plan to mitigate:

1. Not easily human readable
2. How do we version the protocol definitions if it's in SQLite?
3. How do we review changes to the protocol if it's in SQLite?

Not easily human readable
We can add some commands to the powermon CLI to print out full protocols and specific commands. Should be easy since we already convert the protocols to DTOs.

How can we be sure that nothing has changed in the SQLite?
Since the whole DB is checked into git we will be able to see it's changed. I am not sure how human readable the diff will be until we try. We can rely on our integration tests to confirm the protocols act as we expect.

How do we review changes to the protocol if it's in SQLite?
Hopefully the SQLite db is pretty human readable in the git diffs. If it's not we can implement DB version control with the very popular Liquidbase (https://www.liquibase.org/)

@jblance any other concerns?

[jblance]

How does the DB get created?
What happens on update / reinstall of the package?
How do we inherit protocol info eg PI30MAX is just a few changes to PI30
Ive worked on a project that used a DB before and the DB versioning (schema mostly) was a huge pita

[rossandrews]

SQLite doesn't need a server, it's just a file. I will create it and check it in. I don't think we will need to worry about schema db versioning very much.

…

On Tue., Nov. 28, 2023, 12:08 p.m. John Blance, ***@***.***> wrote: How does the DB get created? What happens on update / reinstall of the package? How do we inherit protocol info eg PI30MAX is just a few changes to PI30 Ive worked on a project that used a DB before and the DB versioning (schema mostly) was a huge pita — Reply to this email directly, view it on GitHub <#426 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGYGGPI4LX6SMPVOEJQEPLYGZAFBAVCNFSM6AAAAAA74SNBXKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOJWG42DA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jblance]

what about inheritance?

[rossandrews]

If it's a many-to-many relationship between command definitions and protocols we can use a join table. The command definitions will defined then included in as many protocols as needed. We won't need data level inheritance, we may still want code level inheritance for checksums and such.

For example if Q1 is needed by PI30 and PI30MAX and it's the same for both then it can be linked to both.

[rossandrews]

Do we need to worry about firmware versions? Will the behaviour of running a command change with different levels of firmware?

[jblance]

well, thats another thought.
point 1. pi30 has a bunch of commands (and definitions etc), pi30max is pi30 with some additional commands and some changed commands
point 2. actually point 1 is that pi30 is different for different models of inverter (same for the jkbms protocol), so maybe the 'protocol' concept needs changing