In agriculture, the weed plant identification is a challenging task as it allows farmers to accurately recognize and remove the same plants from their field. In India, conventional methods for detecting and removing weeds require considerable manual labor and skill, resulting in a time-consuming and costly process. With recent advancements in machine learning and computer vision, automated weed detection systems have become more prevalent. We worked an innovative method for crop-weed classification and weed detection that utilizes a Convolutional Neural Network (CNN) to differentiate images of plant into either weed or non-weed categories. The techniques we introduced were developed using an extensive dataset containing 1300 images of sesame (Sesamum indicum L.) crops cultivated in farmlands of China. The proposed approach evaluated on a dataset available on the Roboflow platform. We used ResNet50 architecture for image classification and Faster-RCNN and YOLO (You Only Look Once) for object detection. The YOLOv5 model’s performance was measured by utilizing Precision (P), Recall (R), and the mean Average Precision (mAP) as performance evaluation metrics. The proposed modified YOLOv5 model achieved the best overall performance within the ‘Weeds’ validation subset resulting in a P (80.7), R (81.1), and mAP (86.4). This approach is suitable for bermudagrass, crabgrass and pigweed species of weeds in sesame field. The proposed approach has several practical applications in agriculture, including weed management, crop yield optimization, and environmental sustainability. Furthermore, it has potential use when integrated with other precision farming equipment, making it a cost-effective solution for farmers. We concluded the efficacy of employing deep learning methods for the detection of weed plants and suggest that it has the potential to revolutionize modern agriculture.