Hand gesture is a natural interaction method, and hand gesture recognition is familiar in human–computer interaction. Yet, the variations, as well as the complexity of hand gestures such as self-structural characteristics, views, and illuminations, made hand gesture recognition as a challenging task. Nowadays, the human–computer interaction area enhancement leads to putting interest in the dynamic hand gesture segmentation based on the gesture recognition system. Apart from the lengthy clinical success, dynamic hand gesture segmentation through webcam vision seems challenging due to the light effects, partial occlusion, and complicated environment. Hence, to segment the entire hand gesture region and enhance the segmentation accuracy, this paper develops an improved segmentation and deep learning-based strategy for dynamic hand gesture recognition. The data is gathered from the ISL benchmark dataset that consists of both static as well as dynamic images. The initial process of the proposed model is the pre-processing, which is being performed by grey scale conversion and histogram equalization. Further, the segmentation of gestures is done by the novel Adaptive Hough Transform (AHT), where the theta angle is tuned. Once the segmentation of gestures is done, the optimized Deep Convolutional Neural Network (Deep CNN) is used for gesture recognition. The learning rate, epoch count, and hidden neurons are tuned by the same heuristic concept. As the main contribution, the segmentation and classification are enhanced by the hybridization of Electric Fish Optimization (EFO), and Whale Optimization Algorithm (WOA) called Electric Fish-based Whale Optimization Algorithm (E-WOA). The training of optimized Deep CNN is handled by Dynamic Time Warping (DTW) for avoiding redundant frames, thus enhancing the performance of dynamic hand gestures. Quantitative measurement is accomplished for evaluating hand gesture segmentation and recognition, which portrays the superior behaviour of the proposed model.