Changing instruction language yields upload error

[emseoyk]

• I checked the documentation and related resources and couldn't find an answer to my question.

Your Question

1. I am trying to use RAGAS on Korean dataset. I first tried changing few-shot examples in Korean using how-to guide on the document, which worked perfecly fine. Then I tried manually changing instruction into Korean, which constantly yields error when uploading the result on RAGAS dashboard (Error Message: API Message: An internal server error occurred).
   Interestingly the result csv with scores is saved successfully, but just the upload() doesn't work. I think it has to do with PydanticOutParser, maybe Korean instruction does not match with the predefined keys...
   But it's just my assumption. Do you happen to know why the instruction doesn't work on other languages?
2. Also, is there any way I can save verdicts and reasons for each metric in csv file? As of now I can only get numeric scores when I run result.to_pandas(). I am using context_precision, context_recall, answer_relevancy, answer_correctness, faithfulness for evaluation metrics.

[sahusiddharth]

Hi @emseoyk,

I tried to reproduce the error, but it worked perfectly fine for me. I used the code below. Can you give it a try and let me know if the problem still persists?

from ragas import SingleTurnSample
from ragas.metrics import LLMContextPrecisionWithReference

context_precision = LLMContextPrecisionWithReference(llm=evaluator_llm)

sample = SingleTurnSample(
    user_input="에펠탑은 어디에 위치해 있나요?",  # "Where is the Eiffel Tower located?" in Korean
    reference="에펠탑은 파리에 위치해 있습니다.",  # "The Eiffel Tower is located in Paris." in Korean
    retrieved_contexts=["에펠탑은 파리에 위치해 있습니다."],  # Same context in Korean
)

adapted_prompts = await context_precision.adapt_prompts(language="korean", llm=evaluator_llm)
context_precision.set_prompts(**adapted_prompts)

prompt = context_precision.get_prompts()["context_precision_prompt"]
prompt.instruction += "주어진 질문, 답변 및 컨텍스트를 바탕으로 주어진 답변을 도출하는 데 컨텍스트가 유용했는지 확인하십시오. 유용하다면 '1', 아니라면 '0'으로 판별하여 JSON 출력하십시오."

context_precision.set_prompts(**{"context_precision_prompt": prompt})

result = evaluate(dataset=EvaluationDataset([sample]), metrics=[context_precision])
result.upload()

Output

[emseoyk]

Your code works fine, but when I run mine, I still get the error like below. Is there any way I could print result dict that contains all output like verdict, reason, etc? I want to observe the output and find out why.

{
	"name": "UploadException",
	"message": "Request failed:
API Message: An internal server error occured",
	"stack": "---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File /usr/local/lib/python3.11/site-packages/ragas/sdk.py:84, in check_api_response(response)
     83 try:
---> 84     response.raise_for_status()
     85 except requests.exceptions.HTTPError:

File /usr/local/lib/python3.11/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
   1023 if http_error_msg:
-> 1024     raise HTTPError(http_error_msg, response=self)

HTTPError: 500 Server Error: Internal Server Error for url: https://api.ragas.io/api/v1/alignment/evaluation

During handling of the above exception, another exception occurred:

UploadException                           Traceback (most recent call last)
Cell In[31], line 8
      3 today_date = datetime.now(ZoneInfo('Asia/Seoul')).strftime('%Y%m%d-%H%M')
      5 # df = result_async.to_pandas()
      6 # df.to_csv(f'/volume/HWGI_AI_RAG/evaluation_ragas/ragas_result_{note}.csv')
----> 8 result_async.upload()

File /usr/local/lib/python3.11/site-packages/ragas/dataset_schema.py:542, in EvaluationResult.upload(self, verbose)
    531 root_trace = [
    532     trace for trace in self.ragas_traces.values() if trace.parent_run_id is None
    533 ][0]
    534 packet = json.dumps(
    535     {
    536         \"run_id\": str(root_trace.run_id),
   (...)
    540     cls=ChainRunEncoder,
    541 )
--> 542 response = upload_packet(
    543     path=\"/alignment/evaluation\",
    544     data_json_string=packet,
    545 )
    547 # check status codes
    548 app_url = get_app_url()

File /usr/local/lib/python3.11/site-packages/ragas/sdk.py:54, in upload_packet(path, data_json_string)
     41 base_url = get_api_url()
     43 response = requests.post(
     44     f\"{base_url}/api/v1{path}\",
     45     data=data_json_string,
   (...)
     52     },
     53 )
---> 54 check_api_response(response)
     55 return response

File /usr/local/lib/python3.11/site-packages/ragas/sdk.py:101, in check_api_response(response)
     94     error_msg = f\"\
Status Code: {response.status_code}\"
     96 base_logger.error(
     97     f\"[API_ERROR] Request failed. \"
     98     f\"Status Code: {response.status_code}, URL: {response.url}, \"
     99     f\"Error Message: {error_msg}\"
    100 )
--> 101 raise UploadException(
    102     status_code=response.status_code, message=f\"Request failed: {error_msg}\"
    103 )

UploadException: Request failed:
API Message: An internal server error occured"
}