Development of automatic intelligent video content analysis system is an important goal of both the commercial and public sector surveillance industries. The main tasks in automatic visual surveillance system include moving object detection, static object detection, object classification, object tracking, action recognition and behavior understanding. Objects can be moving or static and can behave normally or abnormally. The abnormality in a scene is also termed as suspicious object, suspicious event, irregular behavior, unusual event, anomaly etc. Detection of suspicious behavior of object involves modeling and classification of actions. Behavior understanding task becomes more and more difficult as complexity of activity increases. Different methodologies studied in this paper identify various problems in video based action recognition system and give direction for further research in this area.