Depression is one of the most common mental health disorders with strong adverse effects on personal and social functioning which can hamper the lives of individuals. The absence of any objective diagnostic aid for depression leads to a range of biases in the diagnosis and ongoing monitoring. This study throws light upon the contribution of gestures and facial points for depression analysis. This paper discusses a novel cumulative video analysis proposed by us based on action units and fictional points for analysis of facial moment. Experimental results are carried out using real world clinical data and interactive sessions with neuro experts. This smart framework developed by us is useful for detection of depression disorders through gesture recognition. The diagnosis is done and appropriate action is taken according to scale of the depression in the patient and the severity of it. © 2015 IEEE.