#1190: Added runtime support for doing golden comparision for flatbuffers in ttrt

[tapspatel]

The goal of this PR is to introduce golden support in runtime. The design was taken into account that other runtimes can plug into MLIR runtime (ttrt is just one possible frontend for mlir runtime). The frontend building off MLIR runtime can register a "callback" function through python, which will return ProgramContext* and Operation* - to be run after every op is executed. The frontend can then call supporting APIs to return currently running data.

• golden information embedded into the flatbuffer as a golden_map
  • key = loc() of the operation
  • value = vector(float)
• ability to dump debug string from both Operation* and it's OpType*
• added Hook callback function to execute after every op
  • python interface with C++ runtime can initialize a callback function that will return ProgramContext* and Operation* void ptrs
  • getOpOutputTensor(programcontext, operation) pybind function support to get a tensor from device for a particular op
  • getOpDebugString(programcontext, operation) pybind function support to get an op debug string
    • this is needed to get the loc() information for the op (which is key in golden map)
• ttrt callback module which does golden comparision against device tensor using pcc
• ttrt golden option which sets a golden function and triggers golden comparision for each op in runtime

Some ops have bad pcc: following up bad pcc ops in separate issues. Parent issue: #1219

[nsmithtt]

Hey Taps, things are looking great! Commented inline for a few API improvements.

[tapspatel]

fixed all the issues. @jnie-TT @nsmithtt Let me know if there's something missing.

@nsmithtt I tried to pybind the tt::target::GoldenTensor but it ended up complaining a ton about mismatch between data type (ie the autogenerated from flatbuffer compiler and what I was trying to pybind). So I have this workaround

.def("get_debug_info_golden", [](tt::runtime::Binary &binary,
                                       std::string &loc) {
        const ::tt::target::GoldenTensor *goldenTensor =
            binary.getDebugInfoGolden(loc);
        if (goldenTensor == nullptr) {
          return std::vector<float>();
        }

        int totalDataSize = std::accumulate((*goldenTensor->shape()).begin(),
                                            (*goldenTensor->shape()).end(), 1,
                                            std::multiplies<int64_t>());
        std::vector<float> dataVec(totalDataSize);
        std::memcpy(dataVec.data(), goldenTensor->data(), totalDataSize);
        return dataVec;

I'm not 100% with this workaround :) however

[nsmithtt]

Hey Taps, it's looking really close! Some more comments inline

[nsmithtt]

This is looking really great! Thank you for being receptive to my pedantry.

[nsmithtt]

Few minor things left, otherwise I think it's good to go!

[tapspatel]

Many pcc issues with several ops resolved :D. Remaining ones

• logical_not
• test_eq
• test_ne
• test_ge
• test_gt
• test_le
• test_lt