In the medical field various motion tracking techniques like block matching, optical flow, and histogram of oriented optical flow (HOOF) are being experimented for the abnormality detection. The information furnished by the existing techniques is inadequate for medical diagnosis. This technique has an inherent drawback, as the entire image is considered for motion vector calculation, increasing the time complexity. Also, the motion vectors of unwanted objects are getting accounted during abnormality detection, leading to misidentification / misdiagnosis. In this research, our main objective is to focus more on the region of abnormality by avoiding the unwanted motion vectors from the rest of the portion of the heart, allowing better time complexity. Proposed a region-based HOOF (RHOOF) for blood motion tracking and estimation; after experimentation, it is observed that RHOOF is four times faster than HOOF. The performance of supervised machine learning techniques was evaluated based on accuracy, precision, sensitivity, specificity, and area under the curve. In the medical field more importance is given to the sensitivity than accuracy. Support vector machine (SVM) has outperformed other technique on sensitivity and time complexity, hence chosen for abnormality classification in this work. An algorithm has been devised to use combination of RHOOF and SVM for the detection of atrial septal defect (ASD). © 2020, Science and Information Organization.