Pavement deterioration prediction is an important task of road inspection mobile robots. Tree-based deep learning models perform well in this prediction but their accuracy remains unsatisfactory. In this paper, to enhance the prediction accuracy of pavement deterioration, we propose a SELECTOR module and Adaptive Genetic Algorithm-based Random Forest Neural Networks (AGA-RFNNs). The SELECTOR module is first used to refine the input data by exploiting correlation analysis, then the RFNNs are designed to make superior accurate and general predictions by learning predictions from random forests, and finally the AGA is applied to train the RFNNs, which considers both the average and maximum fitness of the population and thereby obtaining the optimal solution. Integrating AGA and RFNN, the proposed AGA-RFNN enhances the accuracy of pavement deterioration prediction systems. Experimental results demonstrate that AGA-RFNN outperforms existing models on four deterioration datasets related to cracking, deflection, international roughness index and rutting, respectively.