Welcome to the TABLET github! The goal of this project is to benchmark progress on instruction learning for tabular prediction. Hopefully, we can create models that solve tabular prediction tasks using instructions and few labeled examples.
While many prediction problems require the use of tabular data, often, gathering sufficient training data can be a challenge task, due to costs or privacy issues. Large language models (LLMs) offer considerable world knowledge due to their pre-training and could help improve sample efficiency for these problems. Still, these models are often not completely aligned with many tabular prediction tasks because of model biases from pre-training and lack of information about the task, hurting their performance in the zero and few shot settings.
What if we could use task instructions to help bridge this gap? That’s where TABLET comes in. TABLET is a living benchmark of tabular datasets annotated with task instructions for evaluating how well LLMs utilize instructions for improving performance on tabular prediction tasks.
TABLET is a living benchmark of tabular prediction tasks annotated with instructions. TABLET provides the tools to evaluate models on current tasks and contribute new tasks. The goal is to help researchers develop techniques that improve the sample efficieny of LLMs on tabular prediction.
If TABLET is useful to your work, please cite us.
@article{tabletSlack23,
Author = {Dylan Slack and Sameer Singh},
Title = {TABLET: Learning From Instructions For Tabular Data},
Year = {2023},
journal = {arXiv},
}To download the data, clone the github repository.
git clone https://github.com/dylan-slack/Tablet.gitOnce this completes, the data is stored in this path.
Tablet/data/benchmarkPlease use Python>=3.9. Because of a quirk in one of the packages, please do not use Python=3.9.7. Also, ensure you have pip>=23.0.1.
conda create -n tablet python=3.9.6
conda activate tablet
pip install --upgrade pipIf you want to install the tablet package from source, navigate into the TABLET package directory and install.
cd Tablet
python3 -m pip install -e .Otherwise, you can install from PyPI with pip. [Note: not released yet]
pip install tablet-benchmarkUnfortunately, some naturally occurring instructions come from sources that are not permissively licensed and do not permit hosting elsewhere. We provide a guide for collecting these instructions in
Tablet/fill_missing_instructions.pyOnce this is completed, you can run
python fill_missing_instructions.pyand the instructions will be added to the benchmark data.
The TABLET package offers several useful features for evaluating performance of LLMs + instructions on tabular datasets. TABLET provides code to evaluate arbitrary huggingface models on tasks and also provides tools to simply get the HuggingFace dataset for a particular task so you can perform whatever evaluation you want.
First, let's look at how the task datasets are stored in TABLET. All the tasks are stored in
Tablet/data/benchmark/performanceFor example, the Adult task is store at
Tablet/data/benchmark/performance/AdultWithin this directory, there are different directories for each instruction annotation for the Adult task. For example, let's look at one of the prototypes generated instructions. This instruction is stored at
Tablet/data/benchmark/performance/Adult/prototypes-synthetic-performance-0Instructions collected through other sources have different paths. The rulesets generated instructions all have the directory name
ruleset-synthetic-performance-*And the naturally occurring instructions have
prototypes-naturallanguage-performance-*Note, the usage of prototypes here is just to retain formatting consistency with the other directory names.
Within each directory, there are four files
../test.csv
../test.json
../train.csv
../train.jsonThese are the training and testing sets, stored both in their tabular formats (the .csv's) and their natural language formats (the .json) files. Within the json files, there are each prompt component, like the header, data point serialization, and instruction.
Here's how to use the TABLET package to get a Huggingface dataset for a particular task. Let's say we want to get one of the Adult and Whooping Cough datasets at these locations
Tablet/data/benchmark/performance/Adult/prototypes-synthetic-performance-0
Tablet/data/benchmark/performance/A37/prototypes-synthetic-performance-0We can get the test datasets as follows
from Tablet import evaluate
benchmark_path = "./data/benchmark/performance/"
tasks = ['A37/prototypes-synthetic-performance-0',
'Adult/prototypes-synthetic-performance-0']
evaluator = evaluate.Evaluator(benchmark_path=benchmark_path,
tasks_to_run=tasks,
encoding_format="flan",
k_shot=0)
whooping_cough, adult = evaluator.get_test_hf_datasets()We can specify k_shot here to control how many k_shot instances are sampled from the training data and included into
the prompts. Then, we can access the Adult test data and labels as
test_data, ground_truth_labels = adult['text'], adult['label'] We can also directly evaluate performance on tasks. For instance, evaluating 2-shot Flan-T5 small performance on Adult with prototypes generated instructions with 3 seeds is as follows
from Tablet import evaluate
benchmark_path = "./data/benchmark/performance/"
tasks = ['Adult/prototypes-synthetic-performance-0']
evaluator = evaluate.Evaluator(benchmark_path=benchmark_path,
tasks_to_run=tasks,
encoding_format="flan",
results_file="my_cool_results.txt",
k_shot=2)
evaluator.run_eval(how_many=3)The results will be appended to my_cool_results.txt.
In order to build models that can align themselves with tabular prediction problems extremely well from just instructions and perhaps a few examples, we need many tasks. These are useful for evaluating how well we're doing and could be useful for future supervision.
TABLET makes it easy to create new tasks by writing instructions or generating them with GPT-3 for new datasets. Here's how you do it.
You must have the training and testing for your task stored in pandas df's. Then, you can call Tablet.create. This
function will take care of creating the task for the naturally occuring instructions you provide and will also generate
instructions using GPT-3, if you would like.
from Tablet import create
create.create_task(train_x,
eval_x,
train_y,
eval_y,
name=my_data_set_name,
header="Predict xyz.",
nl_instruction="Generally, people papers are grad students.",
categorical_columns=names_of_categorical_columns,
num=index_of_task,
num_gpt3_revisions=10,
openai_key_path=path_to_open_ai_key,
save_loc="./data/benchmark")Here, train_x and eval_x are the train and test splits. Similarly, train_y and eval_y are the label columns.
This function also accepts the name of the task (e.g., things like Adult or Wine), the header describing the high level goal of the task, and the natural langauge instructions--this is the nl_instructions argument. You must also specify the names of the categorical columns.
The num argument is the index the task with this naturally occurring instruction will be stored under (e.g., prototypes-naturallanguage-performance-{num}).
Further, If you wish to generate instructions with GPT-3, you will need to provide an OpenAI key in a file and give the location of this
file to the openai_key_path argument and specify how many instructions for the prototypes and rulesets templates you
wish to create with num_gpt3_revisions.
To include your awesome new task, please make sure the task's files are under
./data/benchmark/performance/my_new_taskand submit a pull request.
Please also include a short readmd.md in folder describing the goal of the task and the license the data and instructions are under. For instance, something like this is ideal:
Task: Predict how many sheep someone will need to count before they fall asleep.
Data License: Apache 2.0
Instruction License: MITWe'll review it and add it to the benchmark. If you would like your name & website added to the lists of tasks on the homepage, please mention this in the pull request as well.