Skip to content

Latest commit

 

History

History
113 lines (98 loc) · 4.67 KB

README.md

File metadata and controls

113 lines (98 loc) · 4.67 KB
Raw

Kaggle CommonLit Readability Prize 2nd place solution

This is repository of the 2nd place solution of Kaggle CommonLit Readability Prize.
The discription of this solution is available here.
The prediction notebook in the competition is available here.
A partially revised prediction notebook is here.

Hardware

I used three different machines.

  1. Google Cloud Platform
  • Debian 10.10
  • n1-highmem-8 (vCPU x 8, memory 52 GB)
  • 1 x NVIDIA Tesla V100
  1. Google Cloud Platform
  • Ubuntu 18.04
  • a2-highgpu-1g (vCPU x 12, memory 85 GB)
  • 1 x NVIDIA Tesla A100
  1. kaggle notebook

Environment(except for kaggle notebook)

$ docker-compose up --build -d
$ docker exec -it commonlit bash

Data download

Plese download data to ./data from https://www.kaggle.com/c/commonlitreadabilityprize/data and unzip it.

Model download

I used several pretrained models that are publicly available in the kaggle dataset. Please download it to the following directories and unzip it.

Prerocess

Plese download data to ./data from https://www.kaggle.com/takoihiraokazu/commonlit-fold and unzip it. This is the data for cross validation, and is the result of re-running here.

Train

$ sh bin/train.sh

Please note that the results of ex131.py(deberta-v2-xlarge), ex194.py(deberta-v2-xlarge), ex216.py(deberta-v2-xxlarge) and ex407.py(funnel-transformer-large) will change with each train. Also, I did not fix the seed in mlm_roberta_base.py during the competition period, so if you want to reproduce ex237.py, please use this pretrained model instead of output of mlm_roberta_base.py. I have also listed below the hardware numbers where I performed each training during the competition.

exp Hardware
ex014.py 3
ex015.py 3
ex064.py 1
ex072.py 1
ex084.py 1
ex094.py 1
ex107.py 1
ex131.py 2
ex182.py 2
ex190.py 2
ex194.py 2
ex216.py 2
mlm_roberta_base.py 3
ex237.py 3
ex272.py 2
ex292.py 2
ex384.py 2
ex407.py 2
ex429.py 2
ex434.py 2
ex448.py 2
ex450.py 2
ex465.py 2
ex497.py 2
ex507.py 2

Ensemble & Post process

$ sh bin/ensemble_postprocess.sh

The ensemble weights were adjusted based on the output of ensemble.py and by looking at the Public Score. The PostProcess coefficients and thresholds were also adjusted based on the output of postprocess.py and by looking at the Public Score. The final weights, coefficientsare and thresholds as follows. The public and private scores for each model are also listed.

ensebmle

exp model Public Private weight
ex015 roberta base -> svr 0.476 0.478 0.020
ex015 roberta base -> ridge 0.475 0.476 0.020
ex072 roberta large 0.463 0.466 0.088
ex107 bart large 0.469 0.471 0.090
ex182 deberta large 0.460 0.463 0.230
ex190 electra large 0.470 0.471 0.050
ex194 deberta v2 xlarge 0.466 0.467 0.050
ex216 deberta v2 xxlarge 0.465 0.466 0.140
ex237 roberta base(with mlm) 0.476 0.473 0.040
ex272 funnel large base 0.471 0.473 0.050
ex292 mpnet base 0.470 0.473 0.130
ex384 muppet roberta large 0.466 0.468 0.022
ex407 funnel large 0.464 0.471 0.110
ex429 gpt2 medium 0.478 0.482 0.170
ex434 t5 large 0.498 0.494 -0.110
ex448 + ex450 albert xxlarge v2 0.467 0.471 0.120
ex465 electra base 0.482 0.483 -0.170
ex497 bert base uncased 0.506 0.497 -0.140
ex507 distilbart cnn 12 6 0.479 0.477 0.090

post process

  • Predictions >= 0.3 -> Predictions * 1.07
  • 0.3 > Predictions >= 0 -> Predictions * 1.2
  • 0 > Predictions >= -0.7 -> Predictions * 0.97485814
  • -0.7 > Predictions >= -0.9 -> Predictions * 1.01
  • -0.9 > Predictions >= -2.0 -> Predictions * 1.02150304
  • -2.0 > Predictions -> Predictions * 1.02764047

Predict

$ sh bin/predict.sh