We investigate the ability to formulate and solve optimization problems, which is critical across various domains, ranging from operations research and engineering to finance and machine learning, by Open Source Large Language Models.
For more details please see our work: Solving Optimization Problems with Open Source Large Language Model
conda env create -f environment.ymlYou can type python main.py -c [config file] to run the desired prompting technique. For example, to run chain-of-thought + self-consistency:
python main.py -c config/prompt/cot_5shot_self_consistency.json| Technique | Accuracy |
|---|---|
LLaMA-3-8B-Instruct + 5-shot |
57.48% |
LLaMA-3-8B-Instruct + Self-consistency (5-shot, k=15) |
60.81% |
LLaMA-3-8B-Instruct + Chain of Thoughts (5-shot, k=15) |
78.15% |
LLaMA-3-8B-Instruct + Tree of Thoughts (5-shot, b_1 = 3, b_2 = 5) |
62.00% |
DeepSeekCoder-7B-Instruct + Chain of Thoughts (2-shot, k=15) |
71.50% |
DeepSeekCoder-7B-Instruct + Tree of Thoughts (2-shot, b_1 = 3, b_2 = 5) |
61.52% |
DeepSeekCoder-33B-Instruct + Tree of Thoughts (5-shot, b_1 = 1, b_2 = 5) |
62.95% |
| Ensemble Strategy | 80.52% |
If you use our work in your research, please use the following bibtex
@inproceedings{
nguyen2024technical,
title={Technical Report for {ICML} 2024 Automated Math Reasoning Challenge: Solving Optimization Problems with Open Source Large Language Model},
author={Duc M. Nguyen and Sungahn Ko},
booktitle={AI for Math Workshop @ ICML 2024},
year={2024},
url={https://openreview.net/forum?id=4XzGkm1jK0}
}