The importance of accounting for real-world labelling when predicting software vulnerabilities

@article{Jimenez2019TheIO,
  title={The importance of accounting for real-world labelling when predicting software vulnerabilities},
  author={Matthieu Jimenez and Renaud Rwemalika and Mike Papadakis and Federica Sarro and Yves Le Traon and Mark Harman},
  journal={Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  year={2019}
}
  • Matthieu Jimenez, Renaud Rwemalika, M. Harman
  • Published 12 August 2019
  • Computer Science
  • Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability… 

Figures and Tables from this paper

Learning To Predict Vulnerabilities From Vulnerability-Fixes: A Machine Translation Approach
TLDR
This work develops a prediction method using the encoderdecoder framework of machine translation that automatically learns the latent features of code that are linked with vulnerabilities based on the information gained from historical data.
The Impact of Release-based Training on Software Vulnerability Prediction Models
TLDR
The initial findings reveal that the release-based validation approach to vulnerability prediction models leads to lower performance, so there is the need to come up with innovative solutions that can be effectively exploited in real software developing and testing scenarios.
Noisy Label Learning for Security Defects
TLDR
A two-stage learning method based on noise cleaning to identify and remediate the noisy samples, which improves AUC and recall of baselines by up to 8.9% and 23.4% and shows that learning from noisy labels can be effective for data-driven software and security analytics.
Vulnerability Prediction From Source Code Using Machine Learning
TLDR
This study develops a source code representation method that enables us to perform intelligent analysis on the Abstract Syntax Tree form of source code and investigates whether ML can distinguish vulnerable and nonvulnerable code fragments.
Cross-Project Vulnerability Prediction Based on Software Metrics and Deep Learning
TLDR
Analysis of machine learning models constructed, evaluated, and compared based on a dataset of popular real-world PHP software applications indicate that the adoption of software metrics and deep learning may result in vulnerability prediction models with sufficient performance in cross-project vulnerability prediction.
DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning
TLDR
A novel Deep multi-task learning model to automate seven Commit-level Vulnerability Assessment tasks simultaneously based on Common Vulnerability Scoring System metrics is proposed, showing that DeepCVA is the best-performing model with 38% to 59.8% higher Matthews Correlation Coefficient than many supervised and unsupervised baseline models.
A Machine Learning Approach for Vulnerability Curation
TLDR
The design and implementation of a machine learning system to help the curation by by automatically predicting the vulnerability-relatedness of each data item is reported, and there is no uniform ordering of word2vec parameters sensitivity across data sources.
On the Time-Based Conclusion Stability of Software Defect Prediction Models
TLDR
It is shown that depending on which time period the authors evaluate defect predictors, their performance, in terms of F-Score, the area under the curve (AUC), and Mathews Correlation Coefficient (MCC), varies and their results are not consistent.
TRACER: Finding Patches for Open Source Software Vulnerabilities
TLDR
An empirical study is conducted to understand the quality and characteristics of patches for OSS vulnerabilities in two state-of-the-art vulnerability databases and the first automated approach, named TRACER, is proposed, to find patches for an OSS vulnerability from multiple sources.
...
1
2
3
4
...

References

SHOWING 1-10 OF 44 REFERENCES
Vulnerability Prediction Models: A Case Study on the Linux Kernel
TLDR
It is shown that in the context of the Linux kernel, vulnerability prediction models can be superior to random selection and relatively precise, and practitioners have a valuable tool for prioritizing their security inspection efforts.
Challenges with applying vulnerability prediction models
TLDR
This research measures whether vulnerability prediction models built using standard recommendations perform well enough to provide actionable results for engineering resource allocation, and suggests that VPMs must be refined to achieve actionable performance, possibly through security-specific metrics.
Searching for a Needle in a Haystack: Predicting Security Vulnerabilities for Windows Vista
TLDR
A large-scale empirical study on Windows Vista is presented, where the efficacy of classical metrics like complexity, churn, coverage, dependency measures, and organizational structure of the company are evaluated and how well these software measures correlate with vulnerabilities are evaluated.
Is Newer Always Better?: The Case of Vulnerability Prediction Models
TLDR
This paper investigates whether prediction models behave like milk or wine when used to predict future vulnerabilities, and indicates that the recall values are largely in favor of predictors based on older versions.
Can traditional fault prediction models be used for vulnerability prediction?
TLDR
The results suggest that fault prediction models based upon traditional metrics can substitute for specialized vulnerability prediction models, however, both fault prediction andulnerability prediction models require significant improvement to reduce false positives while providing high recall.
Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities
TLDR
This work investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations, and predicted over 80 percent of the known vulnerable files with less than 25 percent false positives for both projects.
Combining Software Metrics and Text Features for Vulnerable File Prediction
TLDR
A novel approach VULPREDICTOR is proposed to predict vulnerable files, it analyzes software metrics and text mining together to build a composite prediction model, which improves the F1 and EffectivenessRatio@20% scores of the best performing state-of-the-art approaches proposed by Walden et al.
[Engineering Paper] Enabling the Continuous Analysis of Security Vulnerabilities with VulData7
TLDR
VulData7 is an extensible framework and dataset of real vulnerabilities, automatically collected from software archives, that retrieves fixes for 1,600 out of the 2,800 reported vulnerabilities of the 4 security critical open source systems.
Predicting vulnerable software components
TLDR
In an investigation of the Mozilla vulnerability history, it was found that components that had a single vulnerability in the past were generally not likely to have further vulnerabilities, however, component that had similar imports or function calls were likely to be vulnerable.
Linear Programming as a Baseline for Software Effort Estimation
TLDR
It is suggested that using LP4EE as a baseline can help reduce conclusion instability and be more accurate and robust than ATLM against different data splits and cross-validation methods for 44% of the cases.
...
1
2
3
4
5
...