Code for the Paper "TravelPlanner: A Benchmark for Real-World Planning with Language Agents".
[Website] • [Paper] • [Dataset] • [Models] • [Leaderboard] • [Environment] • [Twitter]
- 2026/11/18: Release the official structured query annotations for all 1,000 test instances (release). Support local evaluation on both the validation and test sets; see Evaluation.
- 2024/10/23: Release the models finetuned on TravelPlanner. The data could be found here. We use LLama-Factory for fine-tuning.
- 2024/7/14: Support reference information in JSON format.
- 2024/4/21: Provide format check tool for testset submission files. You can run it to check if there are any format errors in your file.
TravelPlanner is a benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints.
For a given query, language agents are expected to formulate a comprehensive plan that includes transportation, daily meals, attractions, and accommodation for each day.
For constraints, from the perspective of real world applications, TravelPlanner includes three types of them: Environment Constraint, Commonsense Constraint, and Hard Constraint.
- Create a conda environment and install dependencies:
conda create -n travelplanner python=3.9
conda activate travelplanner
pip install -r requirements.txt
- Download the database and unzip it to the
TravelPlannerdirectory (i.e.,your/path/TravelPlanner).
In the two-stage mode, language agents are tasked with employing various search tools to gather information. Based on the collected information, language agents are expected to deliver a plan that not only meets the user’s needs specified in the query but also adheres to commonsense constraints.
export OUTPUT_DIR=path/to/your/output/file
# We support MODEL in ['gpt-3.5-turbo-X','gpt-4-1106-preview','gemini','mistral-7B-32K','mixtral']
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
# if you do not want to test google models, like gemini, just input "1".
export GOOGLE_API_KEY=YOUR_GOOGLE_KEY
# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
cd agents
python tool_agents.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME
The generated plan will be stored in OUTPUT_DIR/SET_TYPE.
TravelPlanner also provides an easier mode solely focused on testing their planning ability. The sole-planning mode ensures that no crucial information is missed, thereby enabling agents to focus on planning itself.
Please refer to the paper for more details.
export OUTPUT_DIR=path/to/your/output/file
# We support MODEL in ['gpt-3.5-turbo-X','gpt-4-1106-preview','gemini','mistral-7B-32K','mixtral']
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
# if you do not want to test google models, like gemini, just input "1".
export GOOGLE_API_KEY=YOUR_GOOGLE_KEY
# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
# STRATEGY in ['direct','cot','react','reflexion']
export STRATEGY=direct
cd tools/planner
python sole_planning.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY
In order to parse natural language plans, we use gpt-4 to convert these plans into json formats. We encourage developers to try different parsing prompts to obtain better-formatted plans.
export OUTPUT_DIR=path/to/your/output/file
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
export SET_TYPE=validation
export STRATEGY=direct
# MODE in ['two-stage','sole-planning']
export MODE=two-stage
export TMP_DIR=path/to/tmp/parsed/plan/file
export SUBMISSION_DIR=path/to/your/evaluation/file
cd postprocess
python parsing.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR
# Then these parsed plans should be stored as the real json formats.
python element_extraction.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR
# Finally, combine these plan files for evaluation. We also provide a evaluation example file "example_evaluation.jsonl" in the postprocess folder.
python combination.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --submission_file_dir $SUBMISSION_DIR
The provided evaluation script supports local evaluation on both the validation and test sets. Use the official leaderboard to submit your test-set results.
Prepare a JSONL file with one record per instance in the original dataset order, including idx, query, and plan. Indices start at 1. Include the complete selected split: 180 records for validation or 1,000 records for test. Use "plan": [] for any unselected or unfinished instance. Empty plans do not count as successful. Final Pass Rate is computed over the full split, with a denominator of 180 for validation or 1,000 for test.
# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
export EVALUATION_FILE_PATH=your/evaluation/file/path
cd evaluation
python eval.py --set_type $SET_TYPE --evaluation_file_path $EVALUATION_FILE_PATH
from datasets import load_dataset
# "test" can be substituted by "train" or "validation".
data = load_dataset('osunlp/TravelPlanner','test')['test']
We fine-tune Llama3.1-8B-Instruct and Qwen2-7B-Instruct on TravelPlanner ('sole-planning' mode). The fine-tuned model weights are available on the HuggingFace 🤗.
| Commonsense (Micro) | Commonsense (Macro) | Hard (Micro) | Hard (Macro) | Final Pass Rate | |
|---|---|---|---|---|---|
| Direct Prompting | |||||
| Llama3.1-8B | 60.1 | 0.0 | 7.9 | 2.8 | 0.0 |
| Qwen2-7B | 49.9 | 1.1 | 2.1 | 0.0 | 0.0 |
| Fine-tuning | |||||
| Llama3.1-8B | 78.3 | 17.8 | 19.3 | 6.1 | 3.8 |
| Qwen2-7B | 59.0 | 0.6 | 0.2 | 0.0 | 0.0 |
If you have any problems, please contact Jian Xie, Kai Zhang, Yu Su
If our paper or related resources prove valuable to your research, we kindly ask for citation.
@inproceedings{xie2024travelplanner,
title={TravelPlanner: A Benchmark for Real-World Planning with Language Agents},
author={Xie, Jian and Zhang, Kai and Chen, Jiangjie and Zhu, Tinghui and Lou, Renze and Tian, Yuandong and Xiao, Yanghua and Su, Yu},
booktitle={Forty-first International Conference on Machine Learning},
year={2024}
}
