Remote sensing’s ability to identify small objects is essential for earth observation, but algorithms developed for remote sensing remain challenged in achieving high accuracy. This research addresses critical challenges in remote sensing, including the small size of objects relative to the entire image, random object orientations requiring rotation-invariant detection, and significant variations in object scale. Furthermore, the model effectively handles noise occlusions and demonstrates the effectiveness of the DIoU loss function in improving performance and robustness. This research deals with modifying the YOLOv8 model to improve the detection performance of small object in remote sensing images called EXYOLOv8, which is an amalgamation of DCNC2f, ResBlockCBAM, Head, and DIoU. These modifications include replacing the C2f backbone with DCNC2f to improve feature representation, adding ResBlockCBAM to the neck to enhance the network feature fusion process, and adding one head for detection. In addition, DIoU loss function is more optimal than GIoU, CIoU, SIoU, and WIoU. DIoU proved to be more effective in considering the distance between the prediction box and the ground truth box, especially in the case of small object detection, where the distance becomes more important than the aspect ratio and other parameters. This study compares the performance of the EXYOLOv8 model with YOLOv8-Base and several other object detection models, such as P2-YOLOv8, YOLOv10, and YOLOv11. Experimental results show that EXYOLOv8 results in EXYOLOv8 Exploration1 is 2.199% better than YOLOv8-Base, EXYOLOv8-exploration2 is 2.459% better than YOLOv8-Base.