A vision-based driver assistance system using image processing technology consists of three main modules: road detection, traffic sign detection/identification, obstacle detection. Traffic Sign Recognition is an upcoming research field under applied Computer Vision and Machine Learning which deals with the automatic detection and classification of traffic signs in traffic scene images. In the current paper, we discuss the methodology used to detect the road sign from a traffic sign scene image and present a technique that can be used to construct a system that recognizes road signs in images. Such a system will ensure that each driver is aware of the rules & hazards on the road & will thus help in improving road safety. Also, the location of the road sign in the traffic sign image is unknown. Once these obstacles are overcome, such a system could be integrated in a Driver Assistance System. The primary objective is to develop an algorithm, which will identify various types of road signs from streaming video of real time road scenes in a reasonable time frame. The algorithm has two main parts: the first one, detection which uses the color and shape information to detect a road sign in a traffic scene picture. The second one, prediction & classification based on Neural Network concept. Our algorithm proposes a variety of MATLAB Image Processing Toolbox commands to determine if a road sign is present in current image. If present, the sign is resized and certain features from the Region of Interest (RoI) are fed as a feature vector to a neural net. This trained Neural Classifier then predicts the class of the road sign. Genetic Algorithms (GA) with built-in intelligence could be incorporated into the current system which will try to look for traffic signs only in environments where there is probability of finding/ detecting one. Copyright 2012 ACM.