Pertanika Journal of Science & Technology
Pertanika Journal Home Facebook
Pertanika · Universiti Putra Malaysia Press

Pertanika Journal of Science & Technology

Official journal of Universiti Putra Malaysia for scholarly work across science, engineering and related technologies.

e-ISSN 2231-8526 ISSN 0128-7680
Research article

Evaluating Convolutional Neural Network Architecture for Historical Topographic Hardcopy Maps Analysis: A Study on Training and Validation Accuracy Variation

Saiful Anuar Jaafar, Abdul Rauf Abdul Rasam and Norizan Mat Diah

https://doi.org/10.47836/pjst.32.6.11
KeywordsConvolutional Neural Network (CNN), deep learning, feature map recognition, Historic Topographic Hardcopy Map (HTHM)
Article content

Abstract

Convolutional Neural Networks (CNN) are widely used for image analysis tasks, including object detection, segmentation, and recognition. Given the advanced capability, this study evaluates the effectiveness and performance of CNN architecture for analysing Historical Topographic Hardcopy Maps (HTHM) by assessing variations in training and validation accuracy. The lack of research specifically dedicated to CNN’s application in analysing topographic hardcopy maps presents an opportunity to explore and address the unique challenges associated with this domain. While existing studies have predominantly focused on satellite imagery, this study aims to uncover valuable insights, patterns, and characteristics inherent to HTHM through customised CNN approaches. This study utilises a standard CNN architecture and tests the model’s performance with different epoch settings (20, 40, and 60) using varying dataset sizes (288, 636, 1144, and 1716 images). The results indicate that the optimal operation point for training and validation accuracy is achieved at epoch 40. Beyond epoch 40, the widening gap between training and validation accuracy suggests overfitting. Hence, adding more epochs does not significantly improve accuracy beyond the optimum phase. The experiment also shows that the CNN model obtains a training accuracy of 98%, validation accuracy of 67%, and F1-score overall performance of 77%. The analysis demonstrates that the CNN model performs reasonably well in classifying instances from the HTHM dataset. These findings contribute to a better understanding of the strengths and limitations of the model, providing valuable insights for future research and refinement of classification approaches in the context of topographic hardcopy map analysis.
Supporting literature

References

  1. Ali, L., Alnajjar, F., Jassmi, H. A., Gocho, M., Khan, W., & Serhani, M. A. (2021). Performance evaluation of deep CNN-based crack detection and localization techniques for concrete structures. Sensors, 21(5), Article 1688. https://doi.org/10.3390/s21051688
  2. Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaria, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021). Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data, 8, Article 53. https://doi.org/10.1186/s40537-021-00444-8
  3. Althnian, A., AlSaeed, D., Al-Baity, H., Samha, A., Dris, A., Alzakari, N., Elwafa, A. A., & Kurdi, H. (2021). Impact of dataset size on classification performance: An empirical evaluation in the medical domain. Applied Sciences, 11(2), Article 796. https://doi.org/10.3390/app11020796
  4. Anuar, S., Ibrahim, J., Rauf, A., & Rasam, A. (2021). Towards automated digitization of cartographic hardcopy maps: Reviews of issues, challenges and potentials in Malaysia Library archives. Malaysian Journal of Remote Sensing & GIS, 10(1), 43-51.
  5. Audebert, N., Le Saux, B., & Lefevre, S. (2019). Deep learning for classification of hyperspectral data: A comparative review. IEEE Geoscience and Remote Sensing Magazine, 7(2), 159–173. https://doi.org/10.1109/MGRS.2019.2912563
  6. Bhosle, K., & Musande, V. (2022). Evaluation of CNN model by comparing with convolutional autoencoder and deep neural network for crop classification on hyperspectral imagery. Geocarto International, 37(3), 813–827. https://doi.org/10.1080/10106049.2020.1740950
  7. Barry-Straume, J., Tschannen, A., Engels, D. W., & Fine, E. (2018). An evaluation of training size impact on validation accuracy for optimized convolutional neural networks. SMU Data Science Review, 1(4), Article 12.
  8. Chen, Y., Jiang, H., Li, C., Jia, X., & Ghamisi, P. (2016). Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE transactions on geoscience and remote sensing, 54(10), 6232-6251. https://doi.org/10.1109/TGRS.2016.2584107
  9. Chauhan, R., Ghanshala, K. K., & Joshi, R. C. (2018, December 15-17). Convolutional Neural Network (CNN) for image detection and recognition. [Paper presentation]. First International Conference on Secure Cyber Computing and Communication (ICSCCC), Jalandhar, India. https://doi.org/10.1109/ICSCCC.2018.8703316.
  10. Dawson, H. L., Dubrule, O., & John, C. M. (2023). Impact of dataset size and convolutional neural network architecture on transfer learning for carbonate rock classification. Computers and Geosciences, 171, Article 105284. https://doi.org/10.1016/j.cageo.2022.105284
  11. Dwivedi, D., & Patil, G. (2022). Lightweight convolutional neural network for land use image classification. Journal of Advanced Geospatial Science & Technology, 2(1), 31-48. https://doi.org/10.11113/jagst.v2n1.31
  12. Garbin, C., Zhu, X., & Marques, O. (2020). Dropout vs. batch normalization: An empirical study of their impact to deep learning. Multimedia Tools and Applications 79(19), 12777–12815. https://doi.org/10.1007/s11042-019-08453-9
  13. Guo, W., Yang, W., Zhang, H., & Hua, G. (2018). Geospatial object detection in high resolution satellite images based on multi-scale convolutional neural network. Remote Sensing, 10(1), Article 131. https://doi.org/10.3390/rs10010131
  14. Hamouda, M., Ettabaa, K. S., & Bouhlel, M. S. (2020). Smart feature extraction and classification of hyperspectral images based on convolutional neural networks. IET Image Processing, 14(10), 1999-2005. https://doi.org/10.1049/iet-ipr.2019.1282
  15. Ji, S., Zhang, C., Xu, A., Shi, Y., & Duan, Y. (2018). 3D convolutional neural networks for crop classification with multi-temporal remote sensing images. Remote Sensing, 10(1), Article 75. https://doi.org/10.3390/rs10010075
  16. Johny, A., & Madhusoodanan, K. N. (2021). Dynamic learning rate in deep CNN model for metastasis detection and classification of histopathology images. Computational and Mathematical Methods in Medicine, 2021(1), Article 5557168. https://doi.org/10.1155/2021/5557168
  17. Kandel, I., & Castelli, M. (2020). The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset. ICT Express, 6(4), 312-315. https://doi.org/10.1016/j.icte.2020.04.010
  18. Khalifa, N. E., Loey, M., & Mirjalili, S. (2022). A comprehensive survey of recent trends in deep learning for digital images augmentation. Artificial Intelligence Review, 55(3), 2351-2377. https://doi.org/10.1007/s10462-021-10066-4
  19. Kumar, A., Gaur, N., Chakravarty, S., Alsharif, M. H., Uthansakul, P., & Uthansakul, M. (2024). Analysis of spectrum sensing using deep learning algorithms: CNNs and RNNs. Ain Shams Engineering Journal, 15(3), Article 102505. https://doi.org/10.1016/j.asej.2023.102505
  20. Li, Z., Xin, Q., Sun, Y., & Cao, M. (2021). A deep learning-based framework for automated extraction of building footprint polygons from very high-resolution aerial imagery. Remote Sensing, 13(18), Article 3630 https://doi.org/10.3390/rs13183630
  21. Liu, B., Du, S., & Zhang, X. (2020). Land cover classification using convolutional neural network with remote sensing data and digital surface model. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(3), 39–43. https://doi.org/10.5194/isprs-annals-V-3-2020-39-2020
  22. Poojary, R., Raina, R., & Mondal, A. K. (2021). Effect of data-augmentation on fine-tuned CNN model performance. IAES International Journal of Artificial Intelligence, 10(1), 84-92, https://doi.org/10.11591/ijai.v10.i1.pp84-92
  23. Roslan, N. A. M., Diah, N. M., Ibrahim, Z., Munarko, Y., & Minarno, A. E. (2023). Automatic plant recognition using convolutional neural network on Malaysian medicinal herbs: The value of data augmentation. International Journal of Advances in Intelligent Informatics, 9(1), 136-147. https://doi.org/10.26555/ijain.v9i1.1076
  24. Sharifi, O., Mokhtarzadeh, M., & Beirami, B. A. (2022). A new deep learning approach for classification of hyperspectral images: Feature and decision level fusion of spectral and spatial features in multiscale CNN. Geocarto International, 37(14), 4208-4233. https://doi.org/10.1080/10106049.2021.1882006
  25. Tharwat, A. (2020). Classification assessment methods. Applied Computing and Informatics, 17(1), 168-192. https://doi.org/10.1016/j.aci.2018.08.003
  26. Zeng, P. (2020) On the confusion matrix in credit scoring and its analytical properties. Communications in Statistics-Theory and Methods, 49(9), 2080-2093. https://doi.org/ 10.1080/03610926.2019.1568485