Despite all significant advances in the detection of pedestrians transmitted by computer vision for driver assistance, it remains a challenging issue. Several studies have been conducted to develop the accuracy of pedestrian detection of intelligent surveillance systems. However, person detection in outdoor conditions is a difficult problem due to the variation in lighting, shadows and occlusions. In this paper we have proposed new methodology for pedestrian detection and tracking under varying light condition for advanced driver assistance system. The main aim of the proposed systems is to avoid collision of vehicles with the pedestrians while driving on roads. It is very difficult to detect a pedestrian due to the varying light conditions (night time, fog). With the proposed systems another way of driver provision is realized, which can pay off the weaknesses of the human visual system. The evaluation is done on firsthand dataset that constructed for this purpose as well as on the freely available KAIST multispectral dataset. Three kinds of databases were tested, taking into account various environmental factors with daytime, night time and fog. Experiments shows proposed system is strong and works successfully. © 2020, Springer Nature Singapore Pte Ltd.