In computer vision application object tracking is a challenging problem. Illumination and occlusion are major constraints observed in object tracking. We are focusing on object tracking with partial or full occlusion. Object tracking is done using features like colors and contours. We have proposed a robust Color-based algorithm to track the object and handle occlusion in real time domain. In our algorithm, HSV color model is used and HSV range of the color object to be tracked is selected. Required object area is identified and contour is formed accordingly to track the object. Proposed algorithm is implemented on ARM Cortex-A7 using Open Source Computer Vision (OpenCV) and Linux-embedded platform. The experiment is done using generated real life database and standard unusual crowd activity database. Results showed run time performance, efficiency and accuracy in real time environment. © 2015 IEEE.