A Deep Reinforcement-Based Anomaly Intrusion Detection for Enhancing Network Cybersecurity
Main Article Content
Abstract
Conventional protection methods, such as rules-based firewalls and signature-based detection, are not cutting it in today's environment of increasingly sophisticated and frequent cyberattacks. Cyberattacks nowadays are extremely dynamic and complex, calling for cutting-edge solutions that can change and adapt as the threat does. DRL is an AI subfield that has been successfully addressing difficult decision-making challenges in several fields, including cybersecurity. Here, we make a step forward by using a DRL framework to model cyberattacks; by incorporating real-world events, we make the models more realistic and applicable. We provide a customized approach that greatly improves existing approaches by carefully tailoring DRL (deep reinforcement algorithms to the complex needs of cybersecurity situations, including adversarial training, dynamic environments, bespoke structure of reward and actions, and more.
In this study, we provide an anomaly detection method to detect attacks on network CPS using Deep Reinforcement Learning. Our proposed methodology was tested using several publicly available research datasets to ensure its efficacy.
Downloads
Metrics
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
You are free to:
- Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
- The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
- Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Notices:
You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
References
M. Abdel-Rahman and others, “Advanced cybersecurity measures in IT service operations and their crucial role in safeguarding enterprise data in a connected world,” Eigenpub Review of Science and Technology, vol. 7, no. 1, pp. 138–158, 2023.
A. Heidari and M. A. Jabraeil Jamali, “Internet of Things intrusion detection systems: a comprehensive review and future directions,” Cluster Comput, vol. 26, no. 6, pp. 3753–3780, 2023.
M. Abdulhussein, The Impact of Artificial Intelligence and Machine Learning on Organizations Cybersecurity. Liberty University, 2024.
A. Thakkar and R. Lohiya, “A review on challenges and future research directions for machine learning-based intrusion detection system,” Archives of Computational Methods in Engineering, vol. 30, no. 7, pp. 4245–4269, 2023.
R. I. Mukhamediev et al., “Review of artificial intelligence and machine learning technologies: classification, restrictions, opportunities and challenges,” Mathematics, vol. 10, no. 15, p. 2552, 2022.
Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” IEEE Trans Neural Netw Learn Syst, vol. 33, no. 12, pp. 6999–7019, 2021.
G. Kocher and G. Kumar, “Machine learning and deep learning methods for intrusion detection systems: recent developments and challenges,” Soft comput, vol. 25, no. 15, pp. 9731–9763, 2021.
E. Schiller, A. Aidoo, J. Fuhrer, J. Stahl, M. Ziörjen, and B. Stiller, “Landscape of IoT security,” Comput Sci Rev, vol. 44, p. 100467, 2022.
Y. Bin Zikria, R. Ali, M. K. Afzal, and S. W. Kim, “Next-generation internet of things (iot): Opportunities, challenges, and solutions,” Sensors, vol. 21, no. 4, p. 1174, 2021.
C. P. Kaliappan, K. Palaniappan, D. Ananthavadivel, and U. Subramanian, “Advancing IoT security: a comprehensive AI-based trust framework for intrusion detection,” Peer Peer Netw Appl, pp. 1–21, 2024.
V. Demertzi, S. Demertzis, and K. Demertzis, “An overview of cyber threats, attacks and countermeasures on the primary domains of smart cities,” Applied Sciences, vol. 13, no. 2, p. 790, 2023.
I. A. I. Ahmad, A. C. Anyanwu, S. Onwusinkwue, S. O. Dawodu, O. V. Akagha, and E. Ejairu, “Cybersecurity challenges in smart cities: a case review of African metropolises,” Computer Science & IT Research Journal, vol. 5, no. 2, pp. 254–269, 2024.
M. Sarhan, S. Layeghy, M. Gallagher, and M. Portmann, “From Zero-Shot Machine Learning to Zero-Day Attack Detection. arXiv 2021,” arXiv preprint arXiv:2109.14868.
D. Nair and N. Mhavan, “Augmenting Cybersecurity: A Survey of Intrusion Detection Systems in Combating Zero-day Vulnerabilities,” in Smart Analytics, Artificial Intelligence and Sustainable Performance Management in a Global Digitalised Economy, Emerald Publishing Limited, 2023, pp. 129–153.
S. Applebaum, T. Gaber, and A. Ahmed, “Signature-based and machine-learning-based web application firewalls: a short survey,” Procedia Comput Sci, vol. 189, pp. 359–367, 2021.
M. Alanazi, A. Mahmood, and M. J. M. Chowdhury, “SCADA vulnerabilities and attacks: A review of the state-of-the-art and open issues,” Comput Secur, vol. 125, p. 103028, 2023.
Y. Guo, “A review of Machine Learning-based zero-day attack detection: Challenges and future directions,” Comput Commun, vol. 198, pp. 175–185, 2023.
B. Gao et al., “Enhancing anomaly detection accuracy and interpretability in low-quality and class imbalanced data: A comprehensive approach,” Appl Energy, vol. 353, p. 122157, 2024.
R. Liu, J. Shi, X. Chen, and C. Lu, “Network anomaly detection and security defense technology based on machine learning: A review,” Computers and Electrical Engineering, vol. 119, p. 109581, 2024.
L. Bergman and Y. Hoshen, “Classification-based anomaly detection for general data,” arXiv preprint arXiv:2005.02359, 2020.
R. Jiao et al., “Learning with limited annotations: a survey on deep semi-supervised learning for medical image segmentation,” Comput Biol Med, p. 107840, 2023.
J. Zipfel, F. Verworner, M. Fischer, U. Wieland, M. Kraus, and P. Zschech, “Anomaly detection for industrial quality assurance: A comparative evaluation of unsupervised deep learning models,” Comput Ind Eng, vol. 177, p. 109045, 2023.
M. Z. Zaheer, A. Mahmood, M. H. Khan, M. Segu, F. Yu, and S.-I. Lee, “Generative cooperative learning for unsupervised video anomaly detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 14744–14754.
A. Azab, M. Khasawneh, S. Alrabaee, K.-K. R. Choo, and M. Sarsour, “Network traffic classification: Techniques, datasets, and challenges,” Digital Communications and Networks, vol. 10, no. 3, pp. 676–692, 2024.
G. ALMahadin et al., “VANET network traffic anomaly detection using GRU-based deep learning model,” IEEE Transactions on Consumer Electronics, 2023.
D. Javaheri, S. Gorgin, J.-A. Lee, and M. Masdari, “Fuzzy logic-based DDoS attacks and network traffic anomaly detection methods: Classification, overview, and future perspectives,” Inf Sci (N Y), vol. 626, pp. 315–338, 2023.
R. Al-amri, R. K. Murugesan, M. Man, A. F. Abdulateef, M. A. Al-Sharafi, and A. A. Alkahtani, “A review of machine learning and deep learning techniques for anomaly detection in IoT data,” Applied Sciences, vol. 11, no. 12, p. 5320, 2021.
M. H. Thwaini, “Anomaly Detection in Network Traffic using Machine Learning for Early Threat Detection,” Data and Metadata, vol. 1, p. 72, 2022.
L. I. Khalaf, B. Alhamadani, O. A. Ismael, A. A. Radhi, S. R. Ahmed, and S. Algburi, “Deep Learning-Based Anomaly Detection in Network Traffic for Cyber Threat Identification,” in Proceedings of the Cognitive Models and Artificial Intelligence Conference, 2024, pp. 303–309.
M. Abbasi, A. Shahraki, and A. Taherkordi, “Deep learning for network traffic monitoring and analysis (NTMA): A survey,” Comput Commun, vol. 170, pp. 19–41, 2021.
Z. Yang et al., “A systematic literature review of methods and datasets for anomaly-based network intrusion detection,” Comput Secur, vol. 116, p. 102675, 2022.
A. B. Nassif, M. A. Talib, Q. Nasir, and F. M. Dakalbab, “Machine learning for anomaly detection: A systematic review,” Ieee Access, vol. 9, pp. 78658–78700, 2021.
X. Ma et al., “A comprehensive survey on graph anomaly detection with deep learning,” IEEE Trans Knowl Data Eng, vol. 35, no. 12, pp. 12012–12038, 2021.
D. K. Reddy, H. S. Behera, J. Nayak, P. Vijayakumar, B. Naik, and P. K. Singh, “Deep neural network based anomaly detection in Internet of Things network traffic tracking for the applications of future smart cities,” Transactions on Emerging Telecommunications Technologies, vol. 32, no. 7, p. e4121, 2021.
D. Ageyev, T. Radivilova, O. Mulesa, O. Bondarenko, and O. Mohammed, “Traffic monitoring and abnormality detection methods for decentralized distributed networks,” in Information security technologies in the decentralized distributed networks, Springer, 2022, pp. 287–305.
A. Thakkar and R. Lohiya, “A review on machine learning and deep learning perspectives of IDS for IoT: recent updates, security issues, and challenges,” Archives of Computational Methods in Engineering, vol. 28, no. 4, pp. 3211–3243, 2021.
Z. Ahmad, A. Shahid Khan, C. Wai Shiang, J. Abdullah, and F. Ahmad, “Network intrusion detection system: A systematic study of machine learning and deep learning approaches,” Transactions on Emerging Telecommunications Technologies, vol. 32, no. 1, p. e4150, 2021.
X. Wang et al., “Deep reinforcement learning: A survey,” IEEE Trans Neural Netw Learn Syst, vol. 35, no. 4, pp. 5064–5078, 2022.
V. G. da Silva Ruffo et al., “Anomaly and intrusion detection using deep learning for software-defined networks: A survey,” Expert Syst Appl, p. 124982, 2024.
G. Apruzzese et al., “The role of machine learning in cybersecurity,” Digital Threats: Research and Practice, vol. 4, no. 1, pp. 1–38, 2023.
M. Lydia, G. E. Prem Kumar, and A. I. Selvakumar, “Securing the cyber-physical system: A review,” Cyber-Physical Systems, vol. 9, no. 3, pp. 193–223, 2023.
S. Mohamed and R. Ejbali, “Deep SARSA-based reinforcement learning approach for anomaly network intrusion detection system,” Int J Inf Secur, vol. 22, no. 1, pp. 235–247, 2023.
S. Tharewal, M. W. Ashfaque, S. S. Banu, P. Uma, S. M. Hassen, and M. Shabaz, “Intrusion detection system for industrial Internet of Things based on deep reinforcement learning,” Wirel Commun Mob Comput, vol. 2022, no. 1, p. 9023719, 2022.
B. Sharma, L. Sharma, C. Lal, and S. Roy, “Anomaly based network intrusion detection for IoT attacks using deep learning technique,” Computers and Electrical Engineering, vol. 107, p. 108626, 2023.
A. A. Hammad, S. R. Ahmed, M. K. Abdul-Hussein, M. R. Ahmed, D. A. Majeed, and S. Algburi, “Deep Reinforcement Learning for Adaptive Cyber Defense in Network Security,” in Proceedings of the Cognitive Models and Artificial Intelligence Conference, 2024, pp. 292–297.
A. Malekloo, E. Ozer, M. AlHamaydeh, and M. Girolami, “Machine learning and structural health monitoring overview with emerging technology and high-dimensional data source highlights,” Struct Health Monit, vol. 21, no. 4, pp. 1906–1955, 2022.
S. Thudumu, P. Branch, J. Jin, and J. Singh, “A comprehensive survey of anomaly detection techniques for high dimensional big data,” J Big Data, vol. 7, pp. 1–30, 2020.
M. Naeem et al., “Trends and future perspective challenges in big data,” in Advances in Intelligent Data Analysis and Applications: Proceeding of the Sixth Euro-China Conference on Intelligent Data Analysis and Applications, 15–18 October 2019, Arad, Romania, 2022, pp. 309–325.
I. Lee and B. Perret, “Preparing high school teachers to integrate AI methods into STEM classrooms,” in Proceedings of the AAAI conference on artificial intelligence, 2022, pp. 12783–12791.
G. Aguiar, B. Krawczyk, and A. Cano, “A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework,” Mach Learn, vol. 113, no. 7, pp. 4165–4243, 2024.
S. Latif, H. Cuayáhuitl, F. Pervez, F. Shamshad, H. S. Ali, and E. Cambria, “A survey on deep reinforcement learning for audio-based applications,” Artif Intell Rev, vol. 56, no. 3, pp. 2193–2240, 2023.
X. Yang, E. Howley, and M. Schukat, “ADT: Time series anomaly detection for cyber-physical systems via deep reinforcement learning,” Comput Secur, vol. 141, p. 103825, 2024.
D. Han, “HARNESSING DEEP REINFORCEMENT LEARNING: STUDIES IN ROBOTIC MANIPULATION, ENHANCED SEMANTIC SEGMENTATION, AND SECURING IMAGE CLASSIFIERS,” 2024.
S. Zhang, Y. Li, F. Ye, X. Geng, Z. Zhou, and T. Shi, “A hybrid human-in-the-loop deep reinforcement learning method for UAV motion planning for long trajectories with unpredictable obstacles,” Drones, vol. 7, no. 5, p. 311, 2023.
M. Świechowski, K. Godlewski, B. Sawicki, and J. Mańdziuk, “Monte Carlo tree search: A review of recent modifications and applications,” Artif Intell Rev, vol. 56, no. 3, pp. 2497–2562, 2023.
K. Rajwar, K. Deep, and S. Das, “An exhaustive review of the metaheuristic algorithms for search and optimization: taxonomy, applications, and open challenges,” Artif Intell Rev, vol. 56, no. 11, pp. 13187–13257, 2023.
W. Hilal, S. A. Gadsden, and J. Yawney, “Financial fraud: a review of anomaly detection techniques and recent advances,” Expert Syst Appl, vol. 193, p. 116429, 2022.
M. M. Ali, B. K. Paul, K. Ahmed, F. M. Bui, J. M. W. Quinn, and M. A. Moni, “Heart disease prediction using supervised machine learning algorithms: Performance analysis and comparison,” Comput Biol Med, vol. 136, p. 104672, 2021.
R. Ahmad, R. Wazirali, and T. Abu-Ain, “Machine learning for wireless sensor networks security: An overview of challenges and issues,” Sensors, vol. 22, no. 13, p. 4730, 2022.
R. Sen, G. Heim, and Q. Zhu, “Artificial intelligence and machine learning in cybersecurity: Applications, challenges, and opportunities for mis academics,” Communications of the Association for Information Systems, vol. 51, no. 1, p. 28, 2022.
A. Pinto, L.-C. Herrera, Y. Donoso, and J. A. Gutierrez, “Survey on intrusion detection systems based on machine learning techniques for the protection of critical infrastructure,” Sensors, vol. 23, no. 5, p. 2415, 2023.
A. Piplai, M. Anoruo, K. Fasaye, A. Joshi, T. Finin, and A. Ridley, “Knowledge guided two-player reinforcement learning for cyber attacks and defenses,” in 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), 2022, pp. 1342–1349.
V. Duddu, “A survey of adversarial machine learning in cyber warfare,” Def Sci J, vol. 68, no. 4, p. 356, 2018.
M. A. R. Al Amin, S. Shetty, and C. Kamhoua, “Cyber deception metrics for interconnected complex systems,” in 2022 Winter Simulation Conference (WSC), 2022, pp. 473–483.
H. Rathore, S. K. Sahay, P. Nikam, and M. Sewak, “Robust android malware detection system against adversarial attacks using q-learning,” Information Systems Frontiers, vol. 23, pp. 867–882, 2021.
Y. Huang, L. Huang, and Q. Zhu, “Reinforcement learning for feedback-enabled cyber resilience,” Annu Rev Control, vol. 53, pp. 273–295, 2022.
K. Sethi, Y. V. Madhav, R. Kumar, and P. Bera, “Attention based multi-agent intrusion detection systems using reinforcement learning,” Journal of Information Security and Applications, vol. 61, p. 102923, 2021.
M. Ahsan, K. E. Nygard, R. Gomes, M. M. Chowdhury, N. Rifat, and J. F. Connolly, “Cybersecurity threats and their mitigation approaches using Machine Learning—A Review,” Journal of Cybersecurity and Privacy, vol. 2, no. 3, pp. 527–555, 2022.
M. Ibrahim and R. Elhafiz, “Security analysis of cyber-physical systems using reinforcement learning,” Sensors, vol. 23, no. 3, p. 1634, 2023.
A. Dutta, S. Chatterjee, A. Bhattacharya, and M. Halappanavar, “Deep reinforcement learning for cyber system defense under dynamic adversarial uncertainties,” arXiv preprint arXiv:2302.01595, 2023.
M. Malik and K. S. Saini, “Network Intrusion Detection System using Reinforcement learning,” in 2023 4th International Conference for Emerging Technology (INCET), 2023, pp. 1–4.
C. Fan, M. Chen, X. Wang, J. Wang, and B. Huang, “A review on data preprocessing techniques toward efficient and reliable knowledge discovery from building operational data,” Front Energy Res, vol. 9, p. 652801, 2021.
R. R. dos Santos, E. K. Viegas, A. O. Santin, and V. V Cogo, “Reinforcement learning for intrusion detection: More model longness and fewer updates,” IEEE Transactions on Network and Service Management, vol. 20, no. 2, pp. 2040–2055, 2022.
M. Soltani, K. Khajavi, M. Jafari Siavoshani, and A. H. Jahangir, “A multi-agent adaptive deep learning framework for online intrusion detection,” Cybersecurity, vol. 7, no. 1, p. 9, 2024.
D. N. Singh, S. Jaiswar, P. Jha, V. K. Tiwari, and P. K. Saket, “Adaptive Intrusion Detection Using Deep Reinforcement Learning: A Novel Approach,” International Journal of all Research Educvation & Scientific Methods, vol. 12, no. 05.
N. Jeffrey, Q. Tan, and J. R. Villar, “A hybrid methodology for anomaly detection in Cyber–Physical Systems,” Neurocomputing, vol. 568, p. 127068, 2024.
S. Afrifa, V. Varadarajan, P. Appiahene, T. Zhang, and E. A. Domfeh, “Ensemble machine learning techniques for accurate and efficient detection of botnet attacks in connected computers,” Eng, vol. 4, no. 1, pp. 650–664, 2023.
A. Yazdinejad, M. Kazemi, R. M. Parizi, A. Dehghantanha, and H. Karimipour, “An ensemble deep learning model for cyber threat hunting in industrial internet of things,” Digital Communications and Networks, vol. 9, no. 1, pp. 101–110, 2023.
P. K. Danso, E. C. P. Neto, S. Dadkhah, A. Zohourian, H. Molyneaux, and A. A. Ghorbani, “Ensemble-based intrusion detection for internet of things devices,” in 2022 IEEE 19th International Conference on Smart Communities: Improving Quality of Life Using ICT, IoT and AI (HONET), 2022, pp. 34–39.
H. Zhao, C. Li, X. Yin, X. Li, R. Zhou, and R. Fu, “Ensemble Learning-Enabled Security Anomaly Identification for IoT Cyber–Physical Power Systems,” Electronics (Basel), vol. 11, no. 23, p. 4043, 2022.
N. Jeffrey, Q. Tan, and J. R. Villar, “Using Ensemble Learning for Anomaly Detection in Cyber–Physical Systems,” Electronics (Basel), vol. 13, no. 7, p. 1391, 2024.
E. Vincent, M. Korki, M. Seyedmahmoudian, A. Stojcevski, and S. Mekhilef, “Reinforcement learning-empowered graph convolutional network framework for data integrity attack detection in cyber-physical systems,” CSEE Journal of Power and Energy Systems, 2024.
S. H. Oh, J. Kim, J. H. Nah, and J. Park, “Employing Deep Reinforcement Learning to Cyber-Attack Simulation for Enhancing Cybersecurity,” Electronics (Basel), vol. 13, no. 3, p. 555, 2024.
M. Soltani, K. Khajavi, M. Jafari Siavoshani, and A. H. Jahangir, “A multi-agent adaptive deep learning framework for online intrusion detection,” Cybersecurity, vol. 7, no. 1, p. 9, 2024.
V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, and S. Wermter, “Survey on reinforcement learning for language processing,” Artif Intell Rev, vol. 56, no. 2, pp. 1543–1575, 2023.
L. Zhang and L. Zhang, “Artificial intelligence for remote sensing data analysis: A review of challenges and opportunities,” IEEE Geosci Remote Sens Mag, vol. 10, no. 2, pp. 270–294, 2022.
W. Jia, M. Sun, J. Lian, and S. Hou, “Feature dimensionality reduction: a review,” Complex & Intelligent Systems, vol. 8, no. 3, pp. 2663–2693, 2022.