Hierarchical Interpretable Vision Reasoning Driven Through a Multi-Modal Large Language Model for Depth Estimation

While the most advanced depth estimation methods currently achieve impressive results in ideal weather conditions, they are highly unreliable in adverse weather conditions such as night, rain, and snow. In this paper, we propose a hierarchical interpretable vision reasoning (HIVR) driven through a multimodal large language model (MLLM) that offers a simple and effective solution capable of reliable operation in adverse weather environments. First, we incorporate state-of-the-art image restoration and depth estimation models to construct a callable task model library, known as HIVR. Subsequently, we fine-tune an MLLM to identify weather conditions, automatically generate invocation programs, and drive HIVR by iteratively calling the relevant models until the depth estimation is completed. This approach enables rapid image restoration for different adverse weather conditions, followed by depth estimation, thereby realizing a universal depth estimation model. Additionally, we have utilized GANs to establish a large-scale synthetic KITTI dataset that includes six types of adverse weather conditions. In the comparison with the state-of-the-art models, the results demonstrate that the proposed method achieves an obvious improvement in the comparison with the state-of-the-art models.