​This article presents an innovative approach for detecting visual irregularities in mechanical elements used in railway infrastructure, such as clips, pads, and tracks. Utilizing AI-supported video analysis, we aim to identify abnormalities in these components, proposing a robust system for defect detection, segmentation, and evaluation along the permanent way. A key challenge is the accurate localization of defective sections, expressed in metrics or time, focusing on both Switch Devices and straight segments for comprehensive analysis. Conventional algorithms alone are insufficient for this complexity; therefore, we employ computer vision techniques and ANN for enhanced defect identification. The proposed system also addresses critical safety needs during maintenance and potential track invasions. Furthermore, it can control train speed during maintenance operations and detect obstructions from external elements, enhancing user and worker safety. The current phase emphasizes neural network learning regarding maintenance elements and incorporates Fuzzy logic to facilitate qualitative decision-making, ensuring effective responses to critical situations.