Merge pull request #122 from RMoodsTeam/118-cli-nlp-model-manager

118 cli nlp model manager

nlp/.gitignore

@@ -2,6 +2,7 @@
 .pytest_cache
 .idea/
 models
+_build
 
 # Source folder
 src/__pycache__
@@ -17,6 +18,7 @@ venv/
 
 # File types
 *.json
+!version_models.json
 
 # Environment variables
 .env

nlp/README.Docker.md

@@ -1,22 +1,27 @@
-### Building and running your application
+## Building and running your application
 
-When you're ready, start your application by running:
-`docker compose up --build`.
+When you're ready, build your application by running:
+```docker build -t rmoods-nlp .```
 
-Your application will be available at http://localhost:8000.
+The ```-t``` parameter refers to tag, which is the name of the image. 
+You can replace ``rmoods-nlp`` with any name you like.
 
-### Deploying your application to the cloud
+After the build completes, you can run your application using the following command:
+```docker run --name rmoods-nlp -d rmoods-nlp```
 
-First, build your image, e.g.: `docker build -t myapp .`.
-If your cloud uses a different CPU architecture than your development
-machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
-you'll want to build the image for that platform, e.g.:
-`docker build --platform=linux/amd64 -t myapp .`.
+The ```-d``` parameter runs the container in detached mode, which means the container 
+runs in the background ```--name``` refers to docker name, it can be used 
+instead of container id. 
 
-Then, push it to your registry, e.g. `docker push myregistry.com/myapp`.
+If you want to open docker image console run:
+```docker exec -it rmoods-nlp /bin/bash```
 
-Consult Docker's [getting started](https://docs.docker.com/go/get-started-sharing/)
-docs for more detail on building and pushing.
+```rmoods-nlp``` is the name of the container. If we did not specify a name,
+Docker would have assigned a random name to the container. To check the name of the container, run:
+```docker ps```
 
-### References
+To stop the container, run:
+```docker stop rmoods-nlp```
+
+## References
 * [Docker's Python guide](https://docs.docker.com/language/python/)

nlp/README.md

@@ -1,6 +1,6 @@
 # RMoods NLP Backend
 
-## Using python virtual environment
+### Using python virtual environment
 To create a python virtual environment, run the following command:
 
 ```python -m venv .venv```
@@ -19,12 +19,43 @@ To exit the virtual environment, run the following command:
 
 ```deactivate```
 
-## Running the server
+### Running the server
 To run the server, run the following command:
 
 ```python src/main.py```
 
-## Running the tests
+### Running the tests
 To run the tests, run the following command:
 
 ```python -m pytest```
+
+## Using the models manager
+
+#### To use the models manager, run the following command:
+
+```bash 
+./manager.sh <command> <model_names>
+```
+
+#### Manager commands:
+- remote - Check the status of the models online
+- install - Install the models
+- clean - Delete all models from file system
+- upload - Upload the models to the server
+- status - Check the status of local models with Google Drive
+
+#### Status and remote command marks:
+- &#10060; - Not installed
+- &#9989; - Installed
+- &#10067; - Version mismatch
+
+#### Model names is the optional parameter for upload and install:
+- If no model names are provided, all models will be installed/uploaded
+```bash
+./manager.sh upload 
+```
+- To upload/install model give list of model names, separate them with a space
+```bash
+./manager.sh install sentiment 
+```
+

nlp/docs/index.md

@@ -7,6 +7,7 @@
 installation.md
 using_docker.md
 env_example.md
+using_manager.md
 ```
 
 ```{toctree}

nlp/docs/using_docker.md

@@ -2,28 +2,28 @@
 
 ## Building and running your application
 
-When you're ready, start your application by running:
-```bash
-docker compose up --build .
-```
-Your application will be available at http://localhost:8000.
+When you're ready, build your application by running:
+```docker build -t rmoods-nlp .```
 
-## Deploying your application to the cloud
+The ```-t``` parameter refers to tag, which is the name of the image. 
+You can replace ``rmoods-nlp`` with any name you like.
 
-First, build your image, e.g.: ```bash docker build -t myapp . .``` 
-If your cloud uses a different CPU architecture than your development
-machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
-you'll want to build the image for that platform, e.g.:
+After the build completes, you can run your application using the following command:
+```docker run --name rmoods-nlp -d rmoods-nlp```
 
-```bash 
-docker build --platform=linux/amd64 -t myapp . .
-```
+The ```-d``` parameter runs the container in detached mode, which means the container 
+runs in the background ```--name``` refers to docker name, it can be used 
+instead of container id. 
 
-Then, push it to your registry, e.g. ```bash docker push myregistry.com/myapp .
+If you want to open docker image console run:
+```docker exec -it rmoods-nlp /bin/bash```
 
-Consult Docker's `getting started <https://docs.docker.com/go/get-started-sharing/>`_
-docs for more detail on building and pushing.
+```rmoods-nlp``` is the name of the container. If we did not specify a name,
+Docker would have assigned a random name to the container. To check the name of the container, run:
+```docker ps```
 
-## References
+To stop the container, run:
+```docker stop rmoods-nlp```
 
-* `Docker's Python guide <https://docs.docker.com/language/python/>`_
+## References
+* [Docker's Python guide](https://docs.docker.com/language/python/)

nlp/docs/using_manager.md

@@ -0,0 +1,29 @@
+# Using the models manager
+
+#### To use the models manager, run the following command:
+
+```bash 
+./manager.sh <command> <model_names>
+```
+
+#### Manager commands:
+- remote - Check the status of the models online
+- install - Install the models
+- clean - Delete all models from file system
+- upload - Upload the models to the server
+- status - Check the status of local models with Google Drive
+
+#### Status and remote command marks:
+- &#10060; - Not installed
+- &#9989; - Installed
+- &#10067; - Version mismatch
+
+#### Model names is the optional parameter for upload and install:
+- If no model names are provided, all models will be installed/uploaded
+```bash
+./manager.sh upload 
+```
+- To upload/install model give list of model names, separate them with a space
+```bash
+./manager.sh install sentiment 
+```

nlp/manager.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+command=$1; shift
+array=("$@")
+
+case $command in
+    "install") 
+        if [[ ${#array[@]} -eq 0 ]]; then
+            PYTHONPATH=src python -c 'import version_checker; version_checker.update_model_versions()'
+        else
+            PYTHONPATH=src python -c 'import sys, version_checker; version_checker.update_model_versions(sys.argv[1:])' "${array[@]}"
+        fi
+    ;;
+    "remote")
+        PYTHONPATH=src python -c 'import version_checker; version_checker.get_status()' 
+    ;;
+    "upload")
+        if [[ ${#array[@]} -eq 0 ]]; then
+            PYTHONPATH=src python -c 'import sys, version_checker; version_checker.upload_manager()' 
+        else
+            PYTHONPATH=src python -c 'import sys, version_checker; version_checker.upload_manager(sys.argv[1:])' "${array[@]}"
+        fi
+    ;;
+    "status") 
+        PYTHONPATH=src python -c 'import version_checker; version_checker.get_status(True)'
+    ;;
+    "clean") rm -r models ;;
+    *) echo "Usage: $0 {install|clean|status|remote|upload}" ;;
+esac

nlp/requirements.txt

@@ -9,4 +9,5 @@ google-auth-oauthlib==1.2.1
 tqdm==4.66.5
 myst-parser==4.0.0
 sphinx-code-include==1.4.0
-httplib2==0.22.0
+httplib2==0.22.0
+pytz==2024.2

nlp/src/google_service.py

@@ -5,10 +5,11 @@
 from google.auth.transport.requests import Request
 
 
-def create_service(client_secret_file, api_name, api_version, scopes):
+def create_service(client_secret_file: str, api_name: str, api_version: str,
+                   scopes: list) -> object:
     """
-    Shows basic usage of the Drive v3 API. Based on Google Documentation
-    https://developers.google.com/drive/api/quickstart/python?hl=pl
+    Create a service object for the Google Drive API. Function is based on the Google
+    API documentation: https://developers.google.com/drive/api/quickstart/python?hl=pl
 
     :param client_secret_file: The client secret file.
     :param api_name: The name of the API.
@@ -17,14 +18,8 @@ def create_service(client_secret_file, api_name, api_version, scopes):
 
     :return: The service object or None.
     """
-    CLIENT_SECRET_FILE = client_secret_file
-    API_SERVICE_NAME = api_name
-    API_VERSION = api_version
-    SCOPES = scopes
-
     cred = None
-
-    pickle_file = f'token_{API_SERVICE_NAME}_{API_VERSION}.pickle'
+    pickle_file = f'token_{api_name}_{api_version}.pickle'
 
     if os.path.exists(pickle_file):
         with open(pickle_file, 'rb') as token:
@@ -34,15 +29,15 @@ def create_service(client_secret_file, api_name, api_version, scopes):
         if cred and cred.expired and cred.refresh_token:
             cred.refresh(Request())
         else:
-            flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRET_FILE
-                                                             , SCOPES)
+            flow = InstalledAppFlow.from_client_secrets_file(client_secret_file
+                                                             , scopes)
             cred = flow.run_local_server(port=8002)
 
         with open(pickle_file, 'wb') as token:
             pickle.dump(cred, token)
 
     try:
-        service = build(API_SERVICE_NAME, API_VERSION, credentials=cred)
+        service = build(api_name, api_version, credentials=cred)
         return service
     except Exception as e:
         print('Unable to connect.')