Skip to content

About

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"

Topics

Resources

Stars

573 stars

Watchers

13 watching

Forks

Latest commit

 

History

112 Commits

Folders and files

Repository files navigation

TravelPlanner
A Benchmark for Real-World Planning
with Language Agents

Travel Planner Travel Planner Travel Planner
GPT-4 LLMs


Code for the Paper "TravelPlanner: A Benchmark for Real-World Planning with Language Agents".

Demo Video GIF

[Website] • [Paper] • [Dataset] • [Models] • [Leaderboard] • [Environment] • [Twitter]

Updates

TravelPlanner

TravelPlanner is a benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints.

For a given query, language agents are expected to formulate a comprehensive plan that includes transportation, daily meals, attractions, and accommodation for each day.

For constraints, from the perspective of real world applications, TravelPlanner includes three types of them: Environment Constraint, Commonsense Constraint, and Hard Constraint.

Setup Environment

  1. Create a conda environment and install dependencies:
conda create -n travelplanner python=3.9
conda activate travelplanner
pip install -r requirements.txt
  1. Download the database and unzip it to the TravelPlanner directory (i.e., your/path/TravelPlanner).

Running

Two-stage Mode

In the two-stage mode, language agents are tasked with employing various search tools to gather information. Based on the collected information, language agents are expected to deliver a plan that not only meets the user’s needs specified in the query but also adheres to commonsense constraints.

export OUTPUT_DIR=path/to/your/output/file
# We support MODEL in ['gpt-3.5-turbo-X','gpt-4-1106-preview','gemini','mistral-7B-32K','mixtral']
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
# if you do not want to test google models, like gemini, just input "1".
export GOOGLE_API_KEY=YOUR_GOOGLE_KEY
# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
cd agents
python tool_agents.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME

The generated plan will be stored in OUTPUT_DIR/SET_TYPE.

Sole-Planning Mode

TravelPlanner also provides an easier mode solely focused on testing their planning ability. The sole-planning mode ensures that no crucial information is missed, thereby enabling agents to focus on planning itself.

Please refer to the paper for more details.

export OUTPUT_DIR=path/to/your/output/file
# We support MODEL in ['gpt-3.5-turbo-X','gpt-4-1106-preview','gemini','mistral-7B-32K','mixtral']
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
# if you do not want to test google models, like gemini, just input "1".
export GOOGLE_API_KEY=YOUR_GOOGLE_KEY
# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
# STRATEGY in ['direct','cot','react','reflexion']
export STRATEGY=direct

cd tools/planner
python sole_planning.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY

Postprocess

In order to parse natural language plans, we use gpt-4 to convert these plans into json formats. We encourage developers to try different parsing prompts to obtain better-formatted plans.

export OUTPUT_DIR=path/to/your/output/file
export MODEL_NAME=MODEL_NAME
export OPENAI_API_KEY=YOUR_OPENAI_KEY
export SET_TYPE=validation
export STRATEGY=direct
# MODE in ['two-stage','sole-planning']
export MODE=two-stage
export TMP_DIR=path/to/tmp/parsed/plan/file
export SUBMISSION_DIR=path/to/your/evaluation/file

cd postprocess
python parsing.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR

# Then these parsed plans should be stored as the real json formats.
python element_extraction.py  --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE --tmp_dir $TMP_DIR

# Finally, combine these plan files for evaluation. We also provide a evaluation example file "example_evaluation.jsonl" in the postprocess folder.
python combination.py --set_type $SET_TYPE --output_dir $OUTPUT_DIR --model_name $MODEL_NAME --strategy $STRATEGY --mode $MODE  --submission_file_dir $SUBMISSION_DIR

Evaluation

The provided evaluation script supports local evaluation on both the validation and test sets. Use the official leaderboard to submit your test-set results.

Prepare a JSONL file with one record per instance in the original dataset order, including idx, query, and plan. Indices start at 1. Include the complete selected split: 180 records for validation or 1,000 records for test. Use "plan": [] for any unselected or unfinished instance. Empty plans do not count as successful. Final Pass Rate is computed over the full split, with a denominator of 180 for validation or 1,000 for test.

# SET_TYPE in ['validation', 'test']
export SET_TYPE=validation
export EVALUATION_FILE_PATH=your/evaluation/file/path

cd evaluation
python eval.py --set_type $SET_TYPE --evaluation_file_path $EVALUATION_FILE_PATH

Load Datasets

from datasets import load_dataset
# "test" can be substituted by "train" or "validation".
data = load_dataset('osunlp/TravelPlanner','test')['test']

Model Release

We fine-tune Llama3.1-8B-Instruct and Qwen2-7B-Instruct on TravelPlanner ('sole-planning' mode). The fine-tuned model weights are available on the HuggingFace 🤗.

Commonsense (Micro) Commonsense (Macro) Hard (Micro) Hard (Macro) Final Pass Rate
Direct Prompting
Llama3.1-8B 60.1 0.0 7.9 2.8 0.0
Qwen2-7B 49.9 1.1 2.1 0.0 0.0
Fine-tuning
Llama3.1-8B 78.3 17.8 19.3 6.1 3.8
Qwen2-7B 59.0 0.6 0.2 0.0 0.0

Contact

If you have any problems, please contact Jian Xie, Kai Zhang, Yu Su

Citation Information

If our paper or related resources prove valuable to your research, we kindly ask for citation.

GitHub Stars

@inproceedings{xie2024travelplanner,
  title={TravelPlanner: A Benchmark for Real-World Planning with Language Agents},
  author={Xie, Jian and Zhang, Kai and Chen, Jiangjie and Zhu, Tinghui and Lou, Renze and Tian, Yuandong and Xiao, Yanghua and Su, Yu},
  booktitle={Forty-first International Conference on Machine Learning},
  year={2024}
}

About

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"

Topics

Resources

Stars

573 stars

Watchers

13 watching

Forks

Releases

Packages

Used by

Contributors

Languages