Dynamic imports and processing config

[rossandrews]

Before I continue coding away I wanted to go over the goal behind my refactoring and address some of the concerns you raised with the changes.

A major limitation with the use of dynamic imports is that you don't get to create a custom constructor for each object. This means the objects themselves don't contain the state required to perform their expected operations. So when you call a method on these low state objects a lot of state needs to be included. So this results in a lot state being passed around and the need for the kwargs pattern. The more external state the object depends on the more difficult it is to read it's methods without the context of the code that calls it.

To help reduce the complexity of the code I would like to process all the configuration up front and load it into member variables in the objects. Then the method calls don't require a lot of parameters passed in and will be easier to understand. To accomplish this I will use an object factory pattern that takes in the configuration and returns a complete object. After the model is constructed the configuration will no longer be referenced. This will help with the creation of an API that can change how powermon operates while running.

I do hear your concern about problematic lib imports and I think the importlib idea will still be useful while being used selectively. For example, say a MongoDBOutput type includes a bunch of imports that impacts performance. Within the MongoDBOutput class we can use the importlib to load the Mongo drivers. That way we can still reference the MongoDBOutput class but we only need to install the driver libs if we plan on calling it.

[jblance]

Before I continue coding away I wanted to go over the goal behind my refactoring and address some of the concerns you raised with the changes.

A major limitation with the use of dynamic imports is that you don't get to create a custom constructor for each object. This means the objects themselves don't contain the state required to perform their expected operations. So when you call a method on these low state objects a lot of state needs to be included. So this results in a lot state being passed around and the need for the kwargs pattern. The more external state the object depends on the more difficult it is to read it's methods without the context of the code that calls it.

To help reduce the complexity of the code I would like to process all the configuration up front and load it into member variables in the objects. Then the method calls don't require a lot of parameters passed in and will be easier to understand. To accomplish this I will use an object factory pattern that takes in the configuration and returns a complete object. After the model is constructed the configuration will no longer be referenced. This will help with the creation of an API that can change how powermon operates while running.

I do hear your concern about problematic lib imports and I think the importlib idea will still be useful while being used selectively. For example, say a MongoDBOutput type includes a bunch of imports that impacts performance. Within the MongoDBOutput class we can use the importlib to load the Mongo drivers. That way we can still reference the MongoDBOutput class but we only need to install the driver libs if we plan on calling it.

Im OK with this, there are possibly some complications with this approach especially with the less straight forward components (Im sure there will also be solutions).

Imports can be conditional, and I am keen to minimise absolutely required dependancies whereever possible

I see the powermon code as the chance to improve/refactor as it is not 'released' yet - as an example the yaml config isnt a given, it just seemed like a better approach (expecially compared to the ever increasing command line options)
An important learning from the history of this project is that there is a large variation of what people need, so I am determined to keep flexibility in the design (not that I am suggesting that your approach reduces the flexibility)

[rossandrews]

I've started on the changes, will be a few days before I open a pull request. I am keeping all the config parsing in the constructors and adding abstract parent types for shared configuration. The parent classes is also a good spot to put common functionality, like filters. Haven't got that far yet.

I am also changing the object structure a little bit. For example instead of port and protocol being independent objects I am putting them both into a device object.

Overall the changes will be fairly large. Maybe you can try checking out my branch once I'm done and giving it a test run. Which powermon file are you currently testing with? I can test it before I make the pull request.

[rossandrews]

That should be easy to revert, I only used it since it looked tidy. What version of python do you want to stick to? I'm making good progress on a FastAPI web portal and using the latest version of python for it.

[jblance]

the poetry pyproject.toml has the python version >=3.7, my raspberry pis seem to have as latest 3.9 packaged. 3.11 seems a little new - is there other new (useful) functions other than StrEnum that come with 3.11?
When you are using docker for development, do connect with an interactive shell? Or how do you test commands and see the output?

[rossandrews]

I can connect with the container and run commands but I've only done that to troubleshoot my docker setup. I set an entrypoint in the docker compose file that runs the command. Killing and restarting the containers only takes a few seconds. I'll set my docker to run with python 3.9 and see what breaks.

[jblance]

thanks
Im not really familiar with docker, so was trying to figure out the best way (and document it), I try to have unit tests and run them to make sure I dont break things (though the tests are way behind in real coverage)

[jblance]

I can connect with the container and run commands but I've only done that to troubleshoot my docker setup. I set an entrypoint in the docker compose file that runs the command. Killing and restarting the containers only takes a few seconds. I'll set my docker to run with python 3.9 and see what breaks.

Do you want to PR your current changes so things dont get too far apart?

[jblance]

Hi
Looking more at the code and Ive noted a couple of things

• im trying to get sensible defaults if nothing is supplied, so a powermon command will do something (run once, test port, pi30 protocol etc)
• i think the schedules and commands should be separate with a default schedule available if nothing else is defined

also Ive tried to pull your branch so things dont diverge too much, but im not sure if that has worked - can you try to get your branch updated to master (and if you have suggestions on a better workflow please let me know as it seems hard to incorporate changes)

rough eg of my thinking

scheduling:
  loop_duration: 60
  schedules:
  - name: default  # doesnt need to be defined as exists automatically
    type: once
  - name: every_5_minutes
    type: loop
    run_every_x_loops: 5 #will run every 5 minutes

commands:
  - command: QPIGS
    type: basic #default command type is basic
    schedule: every_5_minutes   #if not defined will default to 'default'
    outputs:
    - name: screen
      tag: Test_Inverter

[rossandrews]

My latest change made it so each schedule only has one command, but I like this approach better so I'll make this change. I've also thought of what to use the command type for. For my case using usbport it will be a "type: poll". For a blue tooth connection we can make "type: continuous" and it won't need to run in a schedule.

[jblance]

sounds good
we'll need to able to add andremove commands from schedules anyway, so this should not complicate things
maybe duration can go, and just have a duration in loop type schdules

scheduling:
  schedules:
  - name: default  # doesnt need to be defined as exists automatically
    type: once
  - name: every_5_minutes
    type: loop
    loop_duration: 300

[rossandrews]

I'm working on the change in the model to separate commands from schedules. FYI, I currently have Monday's set aside to working on this project. Some weeks I find more time but it's inconsistent.

[jblance]

great - try to keep the PRs coming so we dont diverge - we can have broken code on git as long as the pypi packages work