This Repository contains the baselines for evaluation in HUHU shared task at Iberlef 2023
HUHU shared task focuses on examining the use of humor to express prejudice towards minorities, specifically analyzing Spanish tweets that are prejudicial towards:
- Women and feminists
- LGBTIQ community
- Immigrants and racially discriminated people
- Overweight people
Three subtasks are proposed to evaluate this phenomenon
The first subtask consists in determining whether a prejudicial tweet is intended to cause humour. For this, the systems were evaluated and ranked employing the F1-measure over the positive class.
Taking into account the minority groups analyzed, the aim of this subtask is to identify the targeted groups on each tweet as a multilabel classification task. The metric employed for this was macro-F1.
The third subtask consists of predicting on a continuous scale from 1 to 5 to evaluate how prejudicial the message is on average among minority groups. Systems were evaluated employing the Root Mean Squared Error.
To reproduce the results from the proposed baselines you can rely on the python code provided in this repository. Two main paradigms are explored, i.e., the use of classic machine learning (ML) models and neural models, specifically transformer-based. You can use the following command line instruction to train the baselines:
python Baseline.py -mode $PARADIGM -model $MODEL -task $TASK -tf TRAINING_DATA -vf DEV_DATA -phase train
Here PARADIGM variable is used to specify wheter to train a transformer-based system or a classic ML approach, MODEL is used in the case you were using transformers models to specify a pretrained model from hugging face 🤗 library, TASK is used to specify the task you will be training your system.
To obtain more information and see the choices for each variable you can do
python Baseline.py --help
To make predictions you can use
python Baseline.py -model $MODEL -mode $PARADIGM -phase predict -task $TASK -vf DEV_DATA
And for evaluating your predictions:
python Baseline.py -model $MODEL -mode evaluate -gf DEV_DATA -output $PREDS
here PREDS is the path of the fine-tuned model wigths and the previously predicted data.
- dccuchile/bert-base-spanish-wwm-cased
- bigscience/bloom-1b1