カメラ・LiDAR・無線信号を組み合わせて通信ビームを設計する

This paper proposes a multimodal learning framework that integrates camera images, LiDAR point clouds, and radio-frequency (RF) pilots to realize multi-user multiple-input single-output (MU-MISO) beamforming optimization in mixed line-of-sight (LoS) and non-line-of-sight (NLoS) environments. In LoS-dominant environments, beamforming can be designed by utilizing sensing information related to user locations. However, LoS paths cannot be guaranteed to exist for all users in realistic dynamic environments due to blockages. To characterize NLoS propagation, detailed three-dimensional (3D) information of surrounding obstacles is required. This paper integrates camera images and LiDAR points with RF pilots to capture detailed 3D geometric, object-class, and material information of users and surrounding objects. The obtained multimodal information provides useful cues for both direct LoS paths and reflected NLoS propagation, and provides rich channel-related information for beamforming design. Specifically, a bounding-box detection neural network is first used to extract two-dimensional spatial, object-class, and material information from camera images. The image-derived information then guides the selection of LiDAR points to obtain 3D geometric information of users and surrounding obstacles, and assigns object-class and material labels to the selected points. The image-guided LiDAR features, together with RF pilots, are jointly processed by a designed neural architecture to optimize the beamforming vectors. Simulation results show that the proposed method achieves a higher minimum rate than benchmarks.