Skip to content

A Natural Portuguese Language Benchmark (Napolab) for the evaluation of language models.

License

Notifications You must be signed in to change notification settings

ruanchaves/napolab

Repository files navigation

🌎 Natural Portuguese Language Benchmark (Napolab)

The Napolab is your go-to collection of Portuguese datasets for the evaluation of Large Language Models.

📊 Napolab for Large Language Models (LLMs)

A format of Napolab specifically designed for researchers experimenting with Large Language Models (LLMs) is now available. This format includes two main fields:

  • Prompt: The input prompt to be fed into the LLM.
  • Answer: The expected classification output label from the LLM, which is always a number between 0 and 5.

The dataset in this format can be accessed at https://huggingface.co/datasets/ruanchaves/napolab. If you’ve used Napolab for LLM evaluations, please share your findings with us!

Leaderboards

The Open PT LLM Leaderboard incorporates datasets from Napolab.

Guidelines

Napolab adopts the following guidelines for the inclusion of datasets:

  • 🌿 Natural: As much as possible, datasets consist of natural Portuguese text or professionally translated text.
  • Reliable: Metrics correlate reliably with human judgments (accuracy, F1 score, Pearson correlation, etc.).
  • 🌐 Public: Every dataset is available through a public link.
  • 👩‍🔧 Human: Expert human annotations only. No automatic or unreliable annotations.
  • 🎓 General: No domain-specific knowledge or advanced preparation is needed to solve dataset tasks.

Napolab currently includes the following datasets:

assin assin2 rerelem
hatebr reli-sa faquad-nli
porsimplessent

💡 Contribute: We're open to expanding Napolab! Suggest additions in the issues. For more information, read our CONTRIBUTING.md.

🌍 For broader accessibility, all datasets have translations in Catalan, English, Galician and Spanish using the facebook/nllb-200-1.3B model via Easy-Translate.

🤖 Models

We've made several models, fine-tuned on this benchmark, available on Hugging Face Hub:

Datasets mDeBERTa v3 BERT Large BERT Base
ASSIN 2 - STS Link Link Link
ASSIN 2 - RTE Link Link Link
ASSIN - STS Link Link Link
ASSIN - RTE Link Link Link
HateBR Link Link Link
FaQUaD-NLI Link Link Link
PorSimplesSent Link Link Link

For model fine-tuning details and benchmark results, visit EVALUATION.md.

Usage

To reproduce the Napolab benchmark available on the Hugging Face Hub locally, follow these steps:

  1. Clone the repository and install the library:
git clone https://github.com/ruanchaves/napolab.git
cd napolab
pip install -e .
  1. Generate the benchmark file:
from napolab import export_napolab_benchmark, convert_to_completions_format
input_df = export_napolab_benchmark()
output_df = convert_to_completions_format(input_df)
output_df.reset_index().to_csv("test.csv", index=False)

Citation

Our research is ongoing, and we are currently working on describing our experiments in a paper, which will be published soon. In the meanwhile, if you would like to cite our work or models before the publication of the paper, please use the following BibTeX citation for this repository:

@software{Chaves_Rodrigues_napolab_2023,
author = {Chaves Rodrigues, Ruan and Tanti, Marc and Agerri, Rodrigo},
doi = {10.5281/zenodo.7781848},
month = {3},
title = {{Natural Portuguese Language Benchmark (Napolab)}},
url = {https://github.com/ruanchaves/napolab},
version = {1.0.0},
year = {2023}
}

Disclaimer

The HateBR dataset, including all its components, is provided strictly for academic and research purposes. The use of the HateBR dataset for any commercial or non-academic purpose is expressly prohibited without the prior written consent of SINCH.