Infosec scan vulnerabilities preventing deployments

[WVSmith]

There have been quite a few CVE vulnerabilities in python versions lately, which caused our corporate infosec to block deployments of our Triton services. Recent versions of the triton base image handle many of these, especially due to the change to python 3.10 I think it is. What I'm still seeing though is that there is a python version 3.8.3 as part of NSight CLI in the Triton base image. Infosec won't let this pass.

What is the policy on updating various libraries and modules to handle CVE vulnerabilities?
Does anyone have any idea how I can update the NSight CLE python version that is in the Triton base image?

I cannot remove NSight, since it seems to be the mechanism for getting prometheus metrics on GPU performance.

[dyastremsky]

Thanks for asking about this. We do a security scan internally to ensure there are no CVEs in a release. Our software for getting CVEs may differ from the one you use, so they may differ in the CVEs caught and how they are classified. We would need more information on the specific CVEs you are seeing. One option you have is submitting a pull request that fixes the CVE, which we could then review. Once merged, it would be included in the next release.

Does your company work with an NVIDIA Solution Architect or is it part of a program, such as NVIDIA AI Enterprise? That would provide a direct channel for working on your specific needs.

CC: @Christina-Young-NVIDIA

[WVSmith]

Thank you for your response. Even though my company is very large and has been working in data science for a while, I'm the foremost leader in pushing to use tools like Triton Inference Server. I'm not aware of any official relationship with NVidia or one of your Solution Architects.

As for CVEs, I do often just update my triton base image version and it resolves most issues. What I have come to find out though is there is a python 3.8.3 file buried in an NSight folder of the docker image (/usr/local/cuda-12.1/NsightSystems-cli-2023.2.3/target-linux-x64/python/bin/python). It's this python file that is triggering my infosec department. I don't know how to handle this file, nor how important NSight is to the functionality of Triton.

[dyastremsky]

It's possible NSight is included for debugging purposes. You can try removing it and seeing if you have all the functionality you need. Let me know if that ends up resolving your issue.

[rmccorm4]

As David mentioned, I believe the nsight shipped in the container is just to allow for using nsight for debugging/performance profiling. As far as I'm aware, it is not used directly for metrics in Triton, even the GPU metrics. These are queried via DCGM, ex: /usr/lib/x86_64-linux-gnu/libdcgm.so.2.4.7.

As a sanity check, I started the latest 23.08 Triton container, removed the entire nsight directory you mentioned (rm -r /usr/local/cuda-12/NsightSystems-cli-2023.2.3/), and started triton with some example models, ran some inferences, and all metrics (including GPU metrics) were reported correctly as far as I could tell.

As for the actual CVE reported in the nsight-shipped python package, NVAIE would be the official path for addressing this in the container as David mentioned.

[WVSmith]

Thanks, you guys for this help. I've been experimenting with deleting the NSightSystems folder. I think some of the problems of why it looked related to metrics, was that our Cloud Engineering group has not updated the Nvidia and CUDA drivers on these hosts for a while. I need to get them to update to driver 535 so I can properly move to the latest Triton versions.

I guess I really need to start looking into this Nvidia AI Enterprise.