Deep Learning-Based Framework for Zero-Day Attack Detection in Encrypted Network Traffic Using Python

haneen Hussein; Saad  Raad; Ali  Qasim

doi:10.71229/0rxvc265

Authors

Haneen Mohammed Division of Construction and Projects, Mustansiriyah University, 10064 Baghdad, Iraq
Saad Raad Division of Construction and Projects, Mustansiriyah University, 10064 Baghdad, Iraq https://orcid.org/0009-0002-3992-772X
Ali Qasim Computer Science, University of Technology, Baghdad, Iraq

DOI:

https://doi.org/10.71229/0rxvc265

Keywords:

Payload-Agnostic-Intrusion, , Detection,, Flow-Level Statistical Features, CNN–BiLSTM,, Cross-Dataset Evaluation, Domain Shift.

Abstract

The increasing adoption of encryption in modern communication networks has reduced the effectiveness of traditional intrusion detection systems that rely on packet payload inspection. Consequently, there is a growing demand for payload-agnostic intrusion detection approaches capable of analyzing network behavior without accessing packet contents. This study proposes a deep learning-based intrusion detection framework that utilizes flow-level statistical features and a hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN–BiLSTM) architecture to capture both spatial and temporal characteristics of network traffic. The framework was evaluated using the UNSW-NB15 and CIC-IDS2018 benchmark datasets under both in-dataset and cross-dataset evaluation settings. Experimental results demonstrated excellent classification performance in in-dataset experiments, achieving high accuracy and area under the curve (AUC) values. A threshold calibration strategy was further employed to reduce false-positive rates and improve operational reliability. However, cross-dataset evaluation revealed a considerable decline in detection performance, highlighting the impact of domain shift on model generalization across heterogeneous network environments. These findings demonstrate the effectiveness of flow-level statistical features for intrusion detection while emphasizing the importance of developing more robust learning strategies to improve generalization beyond single-dataset training.

References

[1] H. K. Pedarla. (2025). Encrypted Traffic Analytics (ETA): Machine Learning Approaches for Intrusion Detection Without Decryption. International Journal on Science and Technology (IJSAT), 16 (4), 01–10, doi: 10.71097/IJSAT.v16.i4.9559.

[2] S. I. B. Vasquez , P. A. H. Monckton, D. I. L. Munoz, and H. Allende. (2025). Zero-Day Threat Mitigation via Deep Learning in Cloud Environments. Applied Sciences, 15 (14), 01-26, doi: 10.3390/app15147885.

[3] I. A. Alwhbi, C. C. Zou, and R. N. Alharbi. (2024). Encrypted Network Traffic Analysis and Classification Utilizing Machine Learning. Sensors, 24 (11), 01-17, doi: 10.3390/s24113509.

[4] A. Kaissar, A. B. Nassif, and M. N. Injadat. (2022). A Survey on Network Intrusion Detection Using Convolutional Neural Network. In: ITM Web of Conferences, 43: 01003, doi: 10.1051/itmconf/20224301003.

[5] S. Sadhwani, M. A. H. Khan, R. Muthalagu, P. M. Pawar, and K. Suresh. (2026). A Hybrid BiLSTM-CNN Approach for Intrusion Detection for IoT Applications. Scientific Reports, 16: 29079, doi:10.1038/s41598-025-29079-y.

[6] B. Guo, D. Lu, G. Szumel, R. Gui, T. Wang, N. Konz, and M. A. Mazurowski. (2024). The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: An Experimental Study. arXiv preprint, 04368, arXiv:2409.04368.

[7] R. Li, Y. Aierken, Y. Xu, J. Liu, and Y. Tang, (2025). Research on Cross-Dataset Cardiac Signal Domain Generalization and Feature Interpretability. Scientific Reports, 15:33057, doi: 10.1038/s41598-025-33057-9.

[8] D. Canavese, L. Regano, C. Basile, G. Ciravegna, and A. Lioy. (2022). Encryption-agnostic classifiers of traffic originators and their application to anomaly detection. Computers & Security, 113: 102550, doi: 10.1016/j.cose.2021.102550.

[9] J. Chen, L. Song, S. Cai, H. Xie, S. Yin, and B. Ahmad. (2023). TLS-MHSA: An efficient detection model for encrypted malicious traffic based on multi-head self-attention mechanism. ACM Symposium on Applied Computing, Tallinn, 3614014, doi: 10.1145/3613960.3614014.

[10] G. Long, and Z. Zhang. (2023). Deep encrypted traffic detection: An anomaly detection framework based on parallel automatic feature extraction. Security and Communication Networks, 2023: 3316642, doi: 10.1155/2023/3316642.

[11] N. Malekghaini, E. Akbari, M.A. Salahuddin, N. Limam, R. Boutaba, B. Mathieu, S. Moteau, and S. Tuffin. (2023). Deep learning for encrypted traffic classification in the face of data drift: An empirical study. Computer Networks, 229: 109740, doi: 10.1016/j.comnet.2023.109740.

[12] L. Yu, J. Tao, Y. Xu, W. Sun, and Z. Wang. (2024). TLS fingerprint for encrypted malicious traffic detection using session behavior analysis. Computer Networks, 247:110482, doi: 10.1016/j.comnet.2024.110482.

[13] F. Hendaoui, A. Ferchichi, L. Trabelsi, R. Meddeb, R. Ahmed, and M.K. Khelifi. (2024). Advances in deep learning intrusion detection over encrypted data with privacy preservation: a systematic review. Cluster Computing, 27, 8683–8724, doi: 10.1007/s10586-024-04424-4.

[14] Y. Tang, Z. Cai, Y. Zhang, and J. Yu. (2025). Encryptedflow-Itransformer: A sparse temporal modeling framework for intrusion detection on encrypted network traffic. SSRN Electronic Journal, 5357493, doi: 10.2139/ssrn.5357493.

[15] X. Yuan, J. Wan, D. An, and H. Pei. (2025). A novel encrypted traffic detection model based on detachable convolutional GCN-LSTM architecture. Scientific Reports, 15 (1), 01–15, doi: 10.1038/s41598-025-13397-2.

[16] M. Bilal, O. Tariq, and H, Ahmed. (2026). NOS-Gate: Queue-Aware Streaming IDS for Consumer Gateways under Timing-Controlled Evasion. arXiv preprint, 00389, arXiv: 2601.00389.

[17] F. Li, X. Luo, W. Han, B. Fang, and L. Yin. (2026). MTDecipher: Robust encrypted malicious traffic detection via multi-task graph neural networks. Cybersecurity, 9 (1) , 1–19, doi: 10.1186/s42400-025-00522-x.

[18] Z. Xu, Y. Wu, S. Wang, J. Gao, T. Qiu, Z. Wang, H. Wan, amd X. Zhao. (2025). Deep learning-based intrusion detection systems: A survey. arXiv preprint, 2504.07839, doi: 10.48550/arXiv.2504.07839.

[19] M.L. Ali, K. Thakur, S. Schmeelk, J. Debello, and D. Dragos. (2025). Deep learning vs. machine learning for intrusion detection in computer networks: A comparative study. Applied Sciences, 15 (4), 1–22, doi: 10.3390/app15041903.

[20] S.M. Tseng, Y.Q. Wang, and Y. C. Wang. (2024). Multi-class intrusion detection based on transformer for IoT networks using CIC-IoT-2023 dataset. Future Internet, 16 (8), 1–18, doi: 10.3390/fi16080284.

[21] J. A. Shaikh, C. Wang, Saifullah, M.W.U. Sima, M. Arshad, and W.U.A. Rathore. (2025). Memory feedback transformer based intrusion detection system for IoMT healthcare networks. Internet of Things, 25, 1–14, doi: 10.1016/j.iot.2025.101597.

[22] S. Aiswarya, and S. Parthiban, “A CNN and LSTM-based intrusion detection system to enhance IoT security”, in Proceedings of the 2025 International Conference on Information, Implementation, and Innovation in Technology (I2ITCON), Pune, India, 11210625, 2025, doi: 10.1109/I2ITCON65200.2025.11210625.

[23] [23] A. Biyouki, S. Lotfipour, and B. Haghi. (2026). An enhanced deep learning framework for intrusion classification enterprise network using multi-branch CNN-attention architecture. Scientific Reports, 16:34166, doi: 10.1038/s41598-025-34166-1.

[24] G. Apruzzese, L. Pajolay, and M. Conti. (2022). The Cross-evaluation of Machine Learning-based Network Intrusion Detection Systems. IEEE Transactions on Network and Service Management, 19 (4), 3157344, doi: 10.1109/TNSM.2022.3157344.

[25] S.Layeghy, and M. Portmann. (2023). Explainable cross-domain evaluation of ML-based network intrusion detection systems. Computers and Electrical Engineering, 108: 108692, doi: 10.1016/j.compeleceng.2023.108692.

[26] S. Layeghy, M, Baktashmotlagh, and M. Portmann. (2023). DI-NIDS: Domain invariant network intrusion detection system. Knowledge-Based Systems, 273:110626, doi: 10.1016/j.knosys.2023.110626.

[27] C. Zhang, G. Wang, S. Wang, D. Zhan, and M. Yin. (2023). Cross-domain network attack detection enabled by heterogeneous transfer learning. Computer Networks, 227:109692, doi: 10.1016/j.comnet.2023.109692.

[28] M. Cantone, C. Marrocco, and A. Bria. (2024). Machine Learning in Network Intrusion Detection: A Cross-Dataset Generalization Study. IEEE Access, 12, 144489–144508, doi: 10.1109/ACCESS.2024.3472907.

[29] W. Wang, H. Yang, C. Meinel, H.Y. Ozkan, Serna, and C. Mas-Machuca. (2024). Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detection. arXiv preprint, 15382, doi: 10.48550/arXiv.2404.15382.

[30] M.I. Amin, M. Shen, M.K. Ishak, S. Manickam, and S. Karuppayah. (2026). Enhancing generalization of cross-domain intrusion detection. Behaviour & Information Technology, 38, 1360-0494, doi: 10.1080/09540091.2025.2599708.

[31] J. Wilkie, H. Hindy, C. Michie, C.Tachtatzis, J. Irvine, and R. Atkinson. (2026). A Novel Contrastive Loss for Zero-Day Network Intrusion Detection. arXiv preprint, 09902, doi: 10.48550/arXiv.2601.09902.

[32] F. Alotaibi, and S. Maffeis. (2024). Rasd: Semantic Shift Detection and Adaptation for Network Intrusion Detection. in ICT Systems Security and Privacy Protection (SEC 2024). IFIP AICT, 710, 16–30, doi: 10.1007/978-3-031-65175-5_2.

[33] G. Apruzzese, L. Pajolay, and M. Conti. (2022). The Cross-evaluation of Machine Learning-based Network Intrusion Detection Systems. in IEEE Transactions on Network and Service Management, 19 (4) , 1-18, doi: 10.1109/TNSM.2022.3157344.

[34] M. Cantone, C. Marrocco, and A. Bria. (2024). Machine Learning in Network Intrusion Detection: A Cross-Dataset Generalization Study. in IEEE Access, 12, 144489–144508,2024, doi: 10.1109/ACCESS.2024.3472907.

[35] J. Chen, L. Song, S. Cai, H. Xie, S. Yin, and B. Ahmad. (2023). TLS-MHSA: An Efficient Detection Model for Encrypted Malicious Traffic based on Multi-Head Self-Attention Mechanism. Proceedings of the ACM Symposium on Applied Computing, 26:1 – 21, doi: 10.1145/3613960.

[36] X. Yuan, J. Wan, D. An, H. Pei. (2025). A novel encrypted traffic detection model based on detachable convolutional GCN-LSTM. Scientific Reports, 15:13397, doi: 10.1038/s41598-025-13397-2.