Wellcome to Mamo! This is a benchmark for mathematical modeling.
Before you begin, ensure you have met the following requirements:
- You have installed Git.
- You have a GitHub account.
- You have access to the terminal (Command Prompt on Windows, Terminal on macOS and Linux).
- If you want to run Optimization parts, please make sure you have an optimization solver that can read .lp file installed (Here we use COPT, you could use the solver you want and do some small change in the codes)
To get started, clone the repository and install the necessary dependencies:
cd ~/Projects
git clone https://github.com/FreedomIntelligence/Mamo.git
cd Mamo
- For the close source models, we call the APIs, the requirements are:
pip install -r requirements.txt
- For open source models, we use VLLM for deployment and inference.
pip install vllm
Replace the ~/Projects with the directionary for Mamo
- The data consists of two parts: ODE and Optimization, while optimziation part consists of Easy_LP and Complex_LP, see in Data folder (mamo_ode.jsonl, mamo_complex_lp.jsonl, mamo_easy_lp.jsonl)
- Each data is a json object, with
{
"id": id,
"Question": the question,
"Answer": the answer,
"Category": ode/optimization,
"Type": first_order_equation/.../easy_lp/complex_lp
}
To run the benchmark(for close source model calling API), follow these steps:
- Navigate to the Script Directory: First, go to the appropriate script directory. Depending on your needs, use one of the following commands:
cd scripts/scripts_optimization/mamo_script_optimization
or
cd scripts/scripts_ode/mamo_script_ode
-
Fill in the Required Paths Ensure all necessary path variables are correctly set in your environment or script files.
-
Execute the Scripts in Bash: Run the following scripts in order. Choose the appropriate set of scripts based on whether you are working with ODE or optimization:
- For ODE
bash 1.prepare_query_ode.sh
bash 2.run_model_ode.sh
bash 3.run_code_comp_ode.sh
bash 4.fix_error_ode.sh
bash 5.rerun_code_comp_ode.sh
- For Optimization:
bash 1.prepare_query_optimization.sh
bash 2.run_model_optimization.sh
bash 3.run_code_comp_optimization.sh
bash 4.fix_error_optimization.sh
bash 5.rerun_code_comp_optimization.sh
To run the benchmark(for open source model), follow these steps:
- Navigate to the Script Directory: First, go to the appropriate script directory. Depending on your needs, use one of the following commands:
cd scripts/scripts_optimization/mamo_script_optimization
or
cd scripts/scripts_ode/mamo_script_ode
-
Fill in the Required Paths Ensure all necessary path variables are correctly set in your environment or script files.
-
Execute the Scripts in Bash: Run the following scripts in order. Choose the appropriate set of scripts based on whether you are working with ODE or optimization:
- For ODE
bash 1.prepare_query_ode.sh
bash 2.run_model_ode_open.sh
bash 3.run_code_comp_ode.sh
bash 4.fix_error_ode_open.sh
bash 5.rerun_code_comp_ode.sh
- For Optimization:
bash 1.prepare_query_optimization.sh
bash 2.run_model_optimization_open.sh
bash 3.run_code_comp_optimization.sh
bash 4.fix_error_optimization_open.sh
bash 5.rerun_code_comp_optimization.sh
To run the few shots experiment, here is the instruction:
- Navigate to the Script Directory: Select the target file based on your needs and use one of
cd scripts/scripts_optimization/few_shot_scripts_ode
cd scripts/scripts_ode/few_shot_scripts_optimization
-
Fill in the Required Paths: Make sure to update file paths and adjust environment-specific parameters in the scripts in your local environment.
-
Select data from datasets: Extract samples from the corresponding data sets (
mamo_ode.jsonl
,mamo_complex_lp.jsonl
,mamo_easy_lp.jsonl
) to form the data sets (ode_select.jsonl
,lp_select.jsonl
) of few shots. -
Update the appropriate prompts: Choose the prompts with the number of shots you need from data folders
ode_few_shot_prompt
andlp_few_shot_prompt
to update theprompt_template
" in the1.prepare_query_ode.py
and1.prepare_query_optimization.py
respectively. -
Execute the Scripts in Bash: Adopt the scripts sequentially depending on your needs as listed below.
- For ODE
bash 1.prepare_query_ode.sh
bash 2.run_model_ode.sh
bash 3.run_code_comp_ode.sh
bash fix_error_ode.sh
- For Optimization:
bash 1.prepare_query_optimization.sh
bash 2.run_model_optimization.sh
bash 3.run_code_comp_optimization.sh
call remove_quotes.bat
If you use this work, please cite it as follows:
@misc{huang2024mamo,
title={Mamo: a Mathematical Modeling Benchmark with Solvers},
author={Xuhan Huang and Qingning Shen and Yan Hu and Anningzhe Gao and Benyou Wang},
year={2024},
eprint={2405.13144},
archivePrefix={arXiv},
primaryClass={cs.AI}
}