Create session has different gpu memory (EP: CUDA)

[mifikri]

Hi, i'm trying to create session below using c++ and configuring with cmake.
Model : yolov4.onnx

Dependencies:

• Ort : 1.8.2
• Torch: 1.9.0

CMakeLists.txt

cmake_minimum_required(VERSION 3.14)
project(main)

add_executable(${PROJECT_NAME} main.cpp)

target_include_directories(${PROJECT_NAME} 
    PRIVATE 
    /usr/local/include/onnxruntime
)

# find_package(Torch REQUIRED)

target_link_libraries(
    ${PROJECT_NAME}
    onnxruntime
    # ${TORCH_LIBRARIES}
)

main.cpp

#include <onnxruntime_cxx_api.h>
#include <onnxruntime/core/providers/cuda/cuda_provider_factory.h>
#include <unistd.h>
#include <iostream>

void initialize_session(std::string model_path, bool buffered){
    static Ort::Env env(ORT_LOGGING_LEVEL_INFO, "");    
    Ort::SessionOptions session_options;

    Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_options, 0));

    session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_BASIC); 
    session_options.SetExecutionMode(ExecutionMode::ORT_SEQUENTIAL);
    if (buffered) {
        auto session = Ort::Session(env, model_path.c_str(), model_path.size(), session_options);
    }
    else {
        auto session = Ort::Session(env, model_path.c_str(), session_options);
    }
    std::cout<<"model loaded\n";
    
}

int main()
{
    std::string model_path = "/workspaces/vortex-runtime/data/yolov4.onnx";
    bool buffered = false;

    initialize_session(model_path, buffered);
    while(true)
    {
        sleep(1);
        std::cout<<"infer\n";
    }
}

Scenario: cmake configure with other linking libraries (torch)

GPU Memory:

model	Ort 1.8.2	Ort 1.8.2 + Torch 1.9.0
yolov4.onnx	207 MiB	825 MiB

onnxruntime

The profiling test indicate that there is no memory dealocation (cuda free) while create session

onnxruntime + torch

Have anyone ever met these kind of scenario / test ?