Opni Preprocessing Service
This service preprocesses log documents and routes them to corresponding Nats subjects.
- Python
- Preprocess log messages -- by mask words with pre-defined tokens. Fore example,
https://rancher.com-><URL> - Route log messages.
- Workload logs. Identified by workloads watchlist. Send to Nats subject
preprocessed_logs_workload. - control-plane, Rancher and Longhorn logs. Send to Nats subject
preprocessed_logs_pretrained_model.
- Workload logs. Identified by workloads watchlist. Send to Nats subject
| Component | Type | Description |
|---|---|---|
| raw_logs | Nats subject | The logs received from the raw_logs Nats subject come from the ingest plugin. |
| model_workload_parameters | Nats subject | The payload received from the model_workload_parameters Nats subject is a nested JSON dictionary with the workloads of interest selected by the user. This payload is sent by the training controller service when the user updates the watchlist of workloads/ The preprocessing service will then take the payload |
| model-training-parameters | Opensearch index | On startup of service, the preprocessing service will query the Opensearch index model-training-parameters in order to get the last set of workloads from which logs were obtained to train a Deep Learning model. |
| Component | Type | Description |
|---|---|---|
| preprocessed_logs_workload | Nats subject | Once workload logs have been masked, they are published to the preprocessed_logs_workload Nats subject. |
| preprocessed_logs_pretrained_model | Nats subject | Once control-plane, Rancher and Longhorn logs have been masked, they are published to the preprocessed_logs_pretrained_model Nats subject. |
- Unit tests
- Integration tests
- e2e tests
- Manual testing