This paper tactics to develop the novel Tanner-Whitehouse 3 (TW3)-based automated Bone Age Assessment (BAA) model for children with the assistance of Faster Region- Convolutional Neural Network (Faster R-CNN) and Optimal trained Recurrent Neural Network (RNN). The proposed model covers three main stages: (a) Segmentation, (b) Faster R-CNN-based feature learning, and (c) optimal classification. Further, the segmentation of those regions is performed by adaptive Otsu thresholding. A Faster R-CNN is built to learn the features from the segmented regions. Once the features are extracted from the pooling layers of Faster R-CNN, it is subjected to the RNN. As a modification to RNN, the training weight is optimized by the Average Fitness-based Sun Flower Optimization (AF-SFO), and the optimized network predicts the age of the bone. The experimental evaluation of the proposed model over a set of images collected manually and publically shows its superior performance when compared to the state-of-the-art techniques. © 2021 Elsevier Ltd