LLaVA-Tour: A Large Multimodal Model for Japanese Tourist Spot Prediction and Review Generation

Tourist landmark recognition and review generation can significantly enhance travel planning by helping travelers make informed decisions, optimize experiences, and discover new destinations. These capabilities also benefit local economies and improve business efficiency in tourism. The advancements in large multimodal models have demonstrated high performance across a wide range of image processing tasks due to their extensive knowledge and reasoning capabilities. However, there are no large-scale tourism datasets and no models specifically tailored for tourism have been developed. To address these issues, we created a new dataset with over 1.3 million entries from Japanese tourism website Jalan.net, covering three types of tasks: landmark recognition, general and conditioned review generation and one support task: description generation. Using the Large Language-and-Vision Assistant (LLaVA) as the baseline, we developed instruction tuning strategies for these tasks and created a new Large Multimodal Model, LLaVA-Tour. By integrating domain-specific knowledge, our model outperformed state-of-the-art large multimodal models and review generation models in landmark recognition, and general and conditioned review generation. The code and data url are available at https://github.com/HiromasaYamanishi/LLaVATour.