[ISSUE] record.wait_for_feedback_results() with TruLlama not recording results

[paul-gleeson]

Bug Description
I'm using huggingface as the provider to generate feedback from a RAG model that uses TruLlama as the base of the feedback recorder. Even though I'm using record.wait_for_feedback_results(), I'm not seeing any feedback results from the responses of my RAG model. I'm following the same code structure that I used for a LLM response except in the instance I used LLMChain instead of TruLlama.

To Reproduce
Here is my code:

# start a trusession
session = TruSession()
session.reset_database()

# Define evaluation metrics
f_pii_detection_input = Feedback(hugs.pii_detection).on_input()
f_pii_detection_output = Feedback(hugs.pii_detection).on_output()
f_toxicity = Feedback(hugs.toxic).on_input()
f_positive_sentiment = Feedback(hugs.positive_sentiment).on_output()


tru_query_engine_recorder = TruLlama(
    query_engine,
    app_name="LlamaIndex_App",
    app_version="base",
    feedbacks=[f_pii_detection_input, f_pii_detection_output, f_positive_sentiment, f_toxicity],
)


def interact_with_model(prompt_input):
    global tru_recorder
    
    with tru_query_engine_recorder as recording:
        current_timestamp = datetime.now()
        llm_response = query_engine.query(prompt_input)
        
        # Get the record &  extract feedback results
        record = recording.get()
        
        feedback_results_list = [] 
        
        for feedback, result in record.wait_for_feedback_results().items():
            feedback_results_list.append((feedback.name, result.result))
            print(feedback.name, result.result)
        
        # Extract feedback results
        
        pii_input_detected = feedback_results_list[0][1]
        pii_output_detected = feedback_results_list[1][1]
        positive_sentiment = feedback_results_list[2][1]
        toxicity = feedback_results_list[3][1]
        
        with open('llm_responses_eval.csv', mode='a', newline='') as file:
            writer = csv.writer(file)
            writer.writerow([current_timestamp, prompt_input, llm_response, pii_input_detected, pii_output_detected, positive_sentiment, toxicity])
        

    return llm_response

Relevant Logs/Tracebacks
[('pii_detection', None), ('pii_detection', None), ('positive_sentiment', None), ('toxic', None)]

Environment:

• Windows
• Python 3.10

trulens                                   1.1.0
trulens-apps-langchain                    1.0.3
trulens-apps-llamaindex                   1.2.6
trulens-core                              1.2.6
trulens-dashboard                         1.0.2
trulens-feedback                          1.0.2
trulens-providers-huggingface             1.2.6
trulens-providers-litellm                 1.0.2

Additional context
With the current set up the evaluation metrics are working as expected on the LLM.

[paul-gleeson]

Hi @sfc-gh-pdharmana , any idea on the above? thank you!

[sfc-gh-jreini]

Hey @paul-gleeson - I was able to get this working. Here's a slimmed down version of your code I added on to the TruLlama quickstart to test:

os.environ["HUGGINGFACE_API_KEY"] = "hf_..."

from trulens.providers.huggingface import Huggingface

hugs = Huggingface()

f_pii_detection_input = Feedback(hugs.pii_detection).on_input()
f_pii_detection_output = Feedback(hugs.pii_detection).on_output()
f_toxicity = Feedback(hugs.toxic).on_input()
f_positive_sentiment = Feedback(hugs.positive_sentiment).on_output()

tru_query_engine_recorder = TruLlama(
    query_engine,
    app_name="LlamaIndex_App",
    app_version="base",
    feedbacks=[f_pii_detection_input, f_pii_detection_output, f_toxicity, f_positive_sentiment],
)

def interact_with_model(prompt_input):
    
    with tru_query_engine_recorder as recording:
        llm_response = query_engine.query(prompt_input)
        
        # Get the record &  extract feedback results
        record = recording.get()
        
        feedback_results_list = [] 
        
        for feedback, result in record.wait_for_feedback_results().items():
            feedback_results_list.append((feedback.name, result.result))
            print(feedback.name, result.result)
        

    return llm_response

interact_with_model("What did the author do growing up?")

returns:

pii_detection None
pii_detection None
toxic 0.001173359458334744
positive_sentiment 0.08073681592941284
Response(response='The author worked on writing and programming before college.', source_nodes=[NodeWithScore(node=TextNode(id_='539257eb-4a02-4cfc-aad7-85d40024618c', embedding=None, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='de2c782e-de09-486d-8bf2-0908982e5543', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, hash='271b2e239c030737e6dfb9611429a06d72da7d18d4267166902571256fdb6af1'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='2723dd06-2bac-41bb-9c29-aaee09490b1e', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='e99647d69b84b0a13c59268b58406bb00652b41196a27d7b46b56e5d713018ed')}, text="What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.", mimetype='text/plain', start_char_idx=2, end_char_idx=373, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.8244515604176096), NodeWithScore(node=TextNode(id_='98247c03-3991-46d0-89e5-3904525e3f9f', embedding=None, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='de2c782e-de09-486d-8bf2-0908982e5543', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, hash='271b2e239c030737e6dfb9611429a06d72da7d18d4267166902571256fdb6af1'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='c282a241-f83b-464f-9869-72ec55d4b9a3', node_type=<ObjectType.TEXT: '1'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, hash='a5cb3c7be3bf55d03ee09846021df4f2e5a2dba334f1fb86c249d57b0a40cff4'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='cbef1257-76eb-402d-8f27-c5441390b524', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='0b4f822ad12b07d089691e8fe465cfd1a8eb40b43de3358e4fe2ed3606b289e6')}, text='I remember taking the boys to the coast on a sunny day in 2015 and figuring out how to deal with some problem involving continuations while I watched them play in the tide pools. It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England.', mimetype='text/plain', start_char_idx=65997, end_char_idx=66402, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.8235776646037667), NodeWithScore(node=TextNode(id_='41ee0935-30f3-44f8-901f-419edf81e073', embedding=None, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='de2c782e-de09-486d-8bf2-0908982e5543', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, hash='271b2e239c030737e6dfb9611429a06d72da7d18d4267166902571256fdb6af1'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='657b5e04-3da6-47fe-9628-a575f8961fa2', node_type=<ObjectType.TEXT: '1'>, metadata={'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, hash='e0bf148448f13fc91b94f92c03b13147edcc05d6a27ad0c23b93837496c861ce'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='33592b2b-5624-4d04-86aa-0a8381de6825', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='3779908c352fae99eed78144efa64e8a8a6e91708317acae776ad913bbcc456c')}, text="Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been.", mimetype='text/plain', start_char_idx=38502, end_char_idx=38813, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.8190682380704358)], metadata={'539257eb-4a02-4cfc-aad7-85d40024618c': {'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, '98247c03-3991-46d0-89e5-3904525e3f9f': {'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}, '41ee0935-30f3-44f8-901f-419edf81e073': {'file_path': '/Users/jreini/Desktop/development/trulens/examples/quickstart/data/paul_graham_essay.txt', 'file_name': 'paul_graham_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2024-11-16', 'last_modified_date': '2024-11-16'}})