shepherd - shepherd workflow manager #3843
Conversation
|
Yep, you are on the right track here! |
|
FYI, builds are failing though not your fault -- please rebase in order to pick up some recent fixes to the test infrastructure. |
|
Based on our conversation so far, I am liking Shepherd or some variation on that for the name of the tool. |
|
You have the right directory structure for integrating with the rest of the cctools. One addition: In the |
saifulislampi
changed the title
|
I will add the SADE workflow code here. But you are right, it would be difficult to run SADE workflow outside the whole prepared container. Let's brainstorm some other test cases together that we can run on every integration test. Right now, I am using some bash scripts that echo specific lines and transition into different states. |
dthain
changed the title
shepherd/diagram/dot/diagram1.dot
Outdated
| subgraph cluster_service1 { | ||
| node [color=lightblue style=filled] | ||
| color=lightgrey style=filled | ||
| // service1_initialized [label="initialized"] |
There was a problem hiding this comment.
I'd remove entries rather than just comment them.
There was a problem hiding this comment.
Yes. I agree. I will remove all comented code.
shepherd/environment.yml
Outdated
| - graphviz | ||
| - matplotlib | ||
| - seaborn | ||
| # - pip: |
| import sys | ||
|
|
||
|
|
||
| def extract_imports(file_content, source_dir): |
There was a problem hiding this comment.
You may want to use https://docs.python.org/3/library/ast.html rather than grep to get the imports, as regexps are not very robust. For example, I don't think the following case is handled correctly:
if something:
import ...There was a problem hiding this comment.
I used this as a temporary combiner for this project. I will remove this file entirely.
shepherd/readme.md
Outdated
| @@ -0,0 +1,371 @@ | |||
| # Shepherd - Service Orchestration and Monitoring Tool | |||
There was a problem hiding this comment.
We usually use capital letters, like README.md.
shepherd/shepherd.py
Outdated
|
|
||
| preprocess_config(config, filepath) | ||
|
|
||
| print(f"DEBUG: Loaded and preprocessed config from {filepath}") |
There was a problem hiding this comment.
Since this is an independent module, you may want to use the standard python logging module.
shepherd/shepherd.py
Outdated
| for service_name, details in services.items(): | ||
| # Auto-fill log and error files if not specified | ||
| if 'stdout_path' not in details: | ||
| details['stdout_path'] = f"{service_name}_stdout.log" |
There was a problem hiding this comment.
shepherd/shepherd.py
Outdated
| print("DEBUG: Validating and sorting programs") | ||
| required_keys = ['services'] | ||
|
|
||
| for key in required_keys: |
There was a problem hiding this comment.
Any reason why are you checking here and not letting the exception be raised in whatever code uses it?
Also, if you are going to raise it, it is a KeyError.
shepherd/shepherd.py
Outdated
| import time | ||
| import yaml | ||
|
|
||
| # --- config_loader.py --- |
There was a problem hiding this comment.
should these type of comments be removed?
shepherd/shepherd.py
Outdated
| @@ -0,0 +1,438 @@ | |||
| from multiprocessing import Process | |||
There was a problem hiding this comment.
Do you really want to commit this file? It seems to be a concatenation of all the others.
There was a problem hiding this comment.
No. Thanks for pointing it out. I will remove all unncessary files and code update the PR soon.
|
@saifulislampi is it ok to close this PR in favor of the new Shepherd repository? |
Naming Update: "As of now, 'shepherd' has been provisionally selected as the name for this tool." - May 30, 2024
servicexd - Service Execution Daemon [old name]
This is a very early prototype for servicexd - the service execution daemon. The name is currently a placeholder. Please note that I am actively pushing changes to this project, so it may be unstable or broken at times.