ZeroDayBench-22: A Reproducible Protocol and Evaluation Harness for CVE-Linked Source-Code Vulnerability Detection Benchmarks

jenan jader; Musab  Ahmed mohammed

doi:10.71229/69q54g69

Authors

Jenan Jader Msad Department of Information Technology Engineering, Polytechnic College, Al-Furat Al-Awsat Technical University, Karbala 54003, Iraq
Musab Ahmed mohammed Department of English, University of Kirkuk, Iraq

DOI:

https://doi.org/10.71229/69q54g69

Keywords:

vulnerability detection;, benchmark protocol, CVE-linked corpus;, leakage-safe evaluation, source-code security

Abstract

Empirical evaluation of deep-learning source-code vulnerability detectors has been shown to be highly sensitive to how the underlying corpus is constructed and how the train/test split is defined. Recent methodological work has documented that reported accuracies collapse toward chance level once repository leakage, class-style leakage, and duplicate leakage are removed. Rebuilding evaluations under these constraints requires a corpus construction procedure that is transparent, reproducible, and separated from any single detector implementation.

This paper introduces ZeroDayBench-22 as a protocol contribution rather than as a single fixed dataset. The contribution has three parts. First, a reconstruction-oriented benchmark protocol that maps CVE records to specific fix commits, extracts vulnerable and matched non-vulnerable code spans from the same source files, and enforces a repository-disjoint temporal split. Second, a metadata schema that records, per span, the identifiers necessary to reconstruct the exact input from public sources (repository, commit hash, file path, timestamp, language, CWE, label, and normalised-text hash) without redistributing third-party source code. Third, a model-agnostic evaluation harness that takes per-span score files as input and computes ROC-AUC, PR-AUC, Precision, Recall, F1, Recall at fixed FPR operating points, and Recall at top-k, with an explicit sample-size guard that refuses to report Recall@FPR values below the granularity supported by the negative-class count. A prototype metadata release of 240 candidate spans across nine programming languages is documented as an initial reference instance of the protocol. No detector accuracy is reported in this paper; the artefact is the protocol, the schema, and the harness. All code and metadata are released at a public repository under the MIT licence.

References

[1] MITRE Corporation. Common Vulnerabilities and Exposures (CVE) list. https://cve.mitre.org/

[2] National Institute of Standards and Technology NIST. National Vulnerability Database NVD. https://nvd.nist.gov/

[3] MITRE Corporation. Common Weakness Enumeration (CWE). https://cwe.mitre.org/

[4] Kim, S., Woo, S., Lee, H., & Oh, H. (2017). VUDDY: A Scalable Approach for Vulnerable Code Clone Discovery. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 595–614. https://doi.org/10.1109/SP.2017.62

[5] Fan, J., Li, Y., Wang, S., & Nguyen, T. N. (2020). A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR '20), pp. 508–512. https://doi.org/10.1145/3379597.3387501

[6] Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., & Zhong, Y. (2018). VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In Proceedings of the 25th Network and Distributed System Security Symposium (NDSS). https://doi.org/10.14722/ndss.2018.23158

[7] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., & Zhou, M. (2020). CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536–1547. https://doi.org/10.18653/v1/2020.findings-emnlp.139

[8] Bhandari, G., Naseer, A., & Moonen, L. (2021). CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE 2021), pp. 30–39. https://doi.org/10.1145/3475960.3475985

[9] Wartschinski, L., Noller, Y., Vogel, T., Kehrer, T., & Grunske, L. (2022). VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python. Information and Software Technology, 144, 106809. https://doi.org/10.1016/j.infsof.2021.106809

[10] Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2022). Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Transactions on Software Engineering, 48(9), 3280–3296. https://doi.org/10.1109/TSE.2021.3087402

[11] Chakraborty, P., Arumugam, K. K., Alfadel, M., Nagappan, M., & McIntosh, S. (2024). Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets. IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.2024.3423712

[12] Zhou, Y., Liu, S., Siow, J. K., Du, X., & Liu, Y. (2019). Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), pp. 10197–10207.

[13] Li, Y., Wang, S., & Nguyen, T. N. (2021). Vulnerability Detection with Fine-Grained Interpretations. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021), pp. 292–303. https://doi.org/10.1145/3468264.3468597

[14] Chen, Y., Ding, Z., Alowain, L., Chen, X., & Wagner, D. (2023). DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2023), pp. 654–668.

[15] Hindy, H., Atkinson, R., Tachtatzis, C., Colin, J.-N., Bayne, E., & Bellekens, X. (2020). Utilising Deep Learning Techniques for Effective Zero-Day Attack Detection. Electronics, 9(10), 1684. https://doi.org/10.3390/electronics9101684

[16] Sarhan, M., Layeghy, S., Moustafa, N., & Portmann, M. (2021). Towards a standard feature set for network intrusion detection dataset generation. In International Conference on Mobile Networks and Management, Springer.

[17] Ni, D., Yao, H., Weng, T., Gao, G., Wu, Y., & Su, X. (2023). Just-In-Time Defect Prediction on JavaScript Projects: A Replication Study. IEEE Trans. Reliability, 72(3), 1237–1254. https://doi.org/10.1109/TR.2022.3210173