Compression: unable to mount uncompressed logs directory in the container

[BasantaChaulagain]

Hello,
I am unable to compress the log file as the path for uncompressed logs is not mounted in the container. Please see the logs below for further inspection.
CLP starts successfully. uncomp_logs is the <uncompressed_logs-dir>

$ ./clp-package-ubuntu-focal-x86_64-v0.0.1/sbin/start-clp --uncompressed-logs-dir uncomp_logs/
2021-11-02 11:25:29,892 [INFO] [clp] Using default config file at clp-package-ubuntu-focal-x86_64-v0.0.1/etc/clp-config.yaml
2021-11-02 11:25:29,898 [INFO] [clp] Provision docker network bridge
2021-11-02 11:25:30,102 [INFO] [clp] Starting CLP scheduler
2021-11-02 11:25:30,103 [INFO] [clp] Starting scheduler mariadb database
2021-11-02 11:25:33,941 [INFO] [clp] Starting scheduler queue
2021-11-02 11:25:39,687 [INFO] [clp] Initializing scheduler queue
2021-11-02 11:25:41,374 [INFO] [clp] Initializing scheduler database tables
2021-11-02 15:25:41,608 [INFO] Successfully created clp metadata tables for compression and search
2021-11-02 15:25:41,835 [INFO] Successfully created compression_jobs and compression_tasks orchestration tables
2021-11-02 11:25:41,851 [INFO] [clp] Starting scheduler service
2021-11-02 11:25:41,960 [INFO] [clp] Starting CLP worker

Upon compressing, it gives the following error:

$ ./clp-package-ubuntu-focal-x86_64-v0.0.1/sbin/compress uncomp_logs/auth.log 
2021-11-02 11:26:23,597 [INFO] [clp] Using default config file at clp-package-ubuntu-focal-x86_64-v0.0.1/etc/clp-config.yaml
2021-11-02 15:26:23,842 [INFO] [compress] Compression job submitted to compression-job-handler.
2021-11-02 15:26:23,842 [INFO] [compression-job-handler] compression-job-handler started.
2021-11-02 15:26:23,863 [INFO] [job-8] Iterating and partitioning files into tasks.
2021-11-02 15:26:23,863 [ERROR] [job-8] "/opt/clp/uncomp_logs/auth.log" does not exist.
2021-11-02 15:26:23,871 [INFO] [job-8] Waiting for 0 task(s) to finish.

There indeed is no mentioned directory in the container.

root@clp-mini-cluster:/opt/clp# ls   
LICENSE  README.md  bin  etc  lib  requirements-pre-3.7.txt  sbin  var

I checked the permission of the uncompressed-log-directory and it is user:docker. Is there anything else I need to check

[kirkrodrigues]

Hi @BasantaChaulagain,

The bug occurs when passing a relative path to sbin/compress. For now, you could work around it by passing an absolute path instead.

We'll fix it in the next release. Thanks for reporting it!

[BasantaChaulagain]

Hi @kirkrodrigues,

I got another error stating attribute 'inputlist_path' not in the config file.

2021-11-04 14:00:15,761 [INFO] [clp] Using default config file at etc/clp-config.yaml
2021-11-04 18:00:16,127 [INFO] [compress] Compression job submitted to compression-job-handler.
2021-11-04 18:00:16,127 [INFO] [compression-job-handler] compression-job-handler started.
2021-11-04 18:00:16,172 [INFO] [job-16] Iterating and partitioning files into tasks.
2021-11-04 18:00:16,173 [ERROR] [job-16] Exception while processing job-16.
Traceback (most recent call last):
  File "/opt/clp/lib/python3/site-packages/compression_job_handler/compression_job_handler.py", line 269, in handle_job
    with open(pathlib.Path(clp_io_config.inputlist_path).resolve(), 'r') as f:
AttributeError: 'ClpIoConfig' object has no attribute 'inputlist_path'
2021-11-04 18:00:16,173 [ERROR] [job-16] 'ClpIoConfig' object has no attribute 'inputlist_path'

I assume it is reading the config file at the location /opt/.clp-mini-cluster.yaml in the container. Or is there supposed to be a different config file? Please let me know.

[kirkrodrigues]

Looking at the code, line 269 seems to contain a . after input. Whereas the stack-trace doesn't have that .. Is it possible the code was modified somehow?

If not, could you tell me what operating system you extracted the package on?

[BasantaChaulagain]

@kirkrodrigues, No changes have been made in the code after extracting. I used packager on Ubuntu v20.04.3 LTS to build a new package.

[jackluo923]

Hi @BasantaChaulagain , currently, we are unable to reproduce the error you encountered. Can you download the pre-packaged CLP package from this link and perform the compression again?

This package is generated inside a Ubuntu 20.04 docker image. If this package works correctly while the manually generated package has problems, then there may be some environment issue on your side. We can take investigate the environment problem with you. To begin the investigation, please upload the file <package>/lib/python3/site-packages/compression_job_handler/compression_job_handler.py to this issue.

[BasantaChaulagain]

Hi @jackluo923, I tried the pre-built package and it worked. I then manually re-built the package and it worked as well. Perhaps it was due to environment issue on my side. Thanks for your help.