Detection, and mapping of potholes in a precise and punctual manner is an essential task in avoiding road accidents. Today, roadway distresses are manually detected, which requires time and labor. In this paper, we introduce a system which uses deep learning algorithms and is integrated with smartphones to detect potholes in real-time. The user interface of the system is a smartphone application which maps all potholes on a route that the user is traveling. Simultaneously, deep learning object detection algorithm: Single Shot Multi-box Detector (SSD) looks for potholes using a mobile camera in the background. As soon as an unregistered pothole is detected by SSD, coordinates of the pothole are updated to the database in real-time. Accelerometer and gyroscope readings are continuously taken and assessed by a Deep Feed Forward Neural Network model to detect unregistered potholes. This dual mechanism of camera-based as well as accelerometer-gyroscope based detection not only cross validates detections but also provides stable results even if one mechanism fails. The pothole co-ordinates are rendered on the map user interface that can be accessed in the same application. This system with map/navigation feature as front end and two-fold deep learning pothole detection algorithm in backend is an efficient and a zero cost solution for real-time pothole detection.