Heavy vehicles develop technical snag and traffic jam on streets. Accidents between heavy vehicle and road users, for example, pedestrians often result in severe injuries of the weaker street users. The highway safety and traffic jams can be secured with detection of heavy and overloaded vehicles on the highway to facilitate light motor vehicles like cars, scooters. A model for heavy vehicle detection using fine-tuned based on deep learning is proposed to deal with entangled transportation scene. This model comprises two parts, vehicle detection model and vehicle fine-grained detection. This step provides data for the next classification model. Experiments show that vehicle’s make and model can be recognized from transportation images effectively by using our method. Experimental results demonstrate that the proposed detection system performs accurately with other simple and complex scenarios in detecting heavy vehicles in comparison with past vehicle detection systems. © Springer Nature Switzerland AG 2019.