The quality control in the textile manufacturing industry involves identifying fabric defects. The differences in cloth texture and the limited number of damaged samples are also major problems when it comes to the correct identification of faults in fabrics. In most applications, visual inspection is still a vital component of the quality control process. The ability to check surface defects is an essential part of the quality control methods of fabrics and textiles. A fault can be described as whatever interrupts the pattern of fabric repetition. To solve this problem, this paper proposes a supervised learning approach to the detection of fabric flaws. The main purpose of this study will be to design a rotation-invariant fabric defect detection plan that will attain a high detection rate related to current techniques. The study introduces a novel framework for identifying defects in fabrics by fusing a Weighted Bi-directional Feature Pyramid Network (BiFPN) and the Segment Anything Model (SAM), which have been produced via Specific Knowledge Injection. The Weighted BiFPN enhances the fusion of multi-scale features by assigning dynamic weights to the features at various levels, which makes it possible to detect both minor and large-scale flaws in fabric. At the same time, the Segment Anything Model (SAM) is a sophisticated segmentation model that is fine-tuned with knowledge injection to adapt its generic segmentation abilities to the peculiarities of fabric textures and defects. The combination will guarantee accurate localisation and categorisation of the flaws, even under difficult conditions that may include changes in illumination, texture, and type of defects. Experiments carried out on standard textile datasets prove that the suggested BiFPN-SAM framework is considerably more efficient in terms of accuracy, Intersection over Union (IoU), and performance. The fact that the hybrid method can generalise to different faults also underscores its usefulness in textile quality control systems through automated means.