ASCFormer: An Adaptive Strucure-aware Cascaded Transformer for 3D Object Detection

3D object detection has achieved significant progress in outdoor LiDAR point clouds, however, the inherent irregularity and varying sparsity distribution of point occupancy present a key challenge. Existing transformer-based 3D detectors often treat all tokens within the attention window as equally important, regardless of varying sparsity, which not only fails to address the disparities between the varying beam densities but also results in increased memory and computational costs. In this work, we propose an adaptive structure-aware cascaded transformer (ASCFormer) that dynamically captures density-insensitive multiscale structure features to model long-range dependencies via cascaded learning. Our ASCFormer detector includes an adaptive structure-aware token learning module that embeds voxel-level foreground probability and grid-level local density into the grid tokens to enhance structural perception capability. Moreover, we integrate these factors to compute significance scores, which are then utilized in inverse transform sampling to select a subset of multiscale tokens with varying receptive field sizes. To improve the training convergence of the window-based transformer in 3D voxel space, we employ cascaded learning via cross-stage attention to enhance the feature representation capability and refine the localization precision of 3D bounding boxes. This design of structure-aware reweighting effectively enhances the cascade paradigm, making to more adaptable to the varying sparsity distribution of point clouds. Extensive experiments on the KITTI and Waymo Open datasets demonstrate that the proposed ASCFormer detector achieves exceptional performance compared with state-of-the-art 3D object detection methods. The source code is publicly available at https://github.com/Xinglong-Li1/ASCFormer.