Combining 3D structural features and multimodal fusion data to correct illumination effect of plant multispectral point clouds

Plant spectral response and three-dimensional (3D) topological relationships can be obtained by analyzing the plant morphological structure and physiological information contained in the 3D multispectral point cloud of plants through close-range imaging. This information can be used to monitor plant growth dynamics, evaluate plant agronomic traits and provide reference indicators, as well as reveal the interactions among plant phenotypes, genotypes and environmental factors. Currently there is no commercially available and mature sensor which can directly obtain multispectral point cloud data of plants by emitting different wavelengths of laser light. In this study, we proposed a novel method for obtaining single-frame multispectral point clouds of plants based on multimodal image registration. To address the differences in plant multispectral reflectance images caused by illumination inhomogeneity and light intensity difference reflected by plant leaves with different leaf positions, we proposed a method of plant spectral image reflectance correction combining 3D structural features and artificial neural network (ANN) models. The results showed that compared with some classical image registration algorithms tested in this study, the proposed joint registration algorithm had a better registration performance for plant RGB images and multispectral images acquired by different sensors, with an average structural similarity (SSIM) of 0.931, which was better than the average registration performance of other classical algorithms (SSIM=0.889). The whiteboard reflectance distribution at different positions and orientations was simulated with high accuracy using ANN, where R-square (R²) and root mean squared error (RMSE) was 0.962 and 0.036, respectively. Compared with the ground truth, the average RMSE of the spectra before and after correction at different leaf positions decreased from 0.182 to 0.040, a decrease of 78.0%; for the same leaf position, the average value of the range (Max-Min) of Euclidean distances between pairs of spectral reflectance curves measured from multiple viewing angles decreased from 0.140 to 0.055, a decrease of 60.7%. The proposed method of combining 3D structure features for correction of multispectral images in this study had good optimization performance for proximally acquired plant multispectral images.