Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes

  title={Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes},
  author={Daniel Votipka and Rock Stevens and Elissa M. Redmiles and Jeremy Hu and Michelle L. Mazurek},
  journal={2018 IEEE Symposium on Security and Privacy (SP)},
Identifying security vulnerabilities in software is a critical task that requires significant human effort. [] Key Result The results suggest that hackers and testers follow similar processes, but get different results due largely to differing experiences and therefore different underlying knowledge of security concepts.

Figures and Tables from this paper

Who are Vulnerability Reporters?: A Large-scale Empirical Study on FLOSS

This paper presents what is believed to be the first large-scale empirical study on the people and organizations who report vulnerabilities in popular FLOSS projects, and investigates several aspects of the vulnerability discovery process, specifically regarding the distribution of contributions, their temporal characteristics, and the motivations of reporters.

Understanding security mistakes developers make: Qualitative analysis from Build It, Break It, Fix It

This paper investigates how and why programmers, despite a baseline of security experience, make security-relevant errors, and conducts an in-depth analysis of 94 submissions to a secure-programming contest designed to mimic real-world constraints: correctness, performance, and security.

Examining Penetration Tester Behavior in the Collegiate Penetration Testing Competition

This work constructed 98 timelines of vulnerability discovery and exploits for 37 unique vulnerabilities discovered by 10 teams of penetration testers and grouped related vulnerabilities together by mapping to Common Weakness Enumerations and MITRE ATT&CK™.

A Case Study of Software Security Red Teams at Microsoft

The results of this work are applicable to practitioners, researchers, and toolsmiths who wish to understand how offensive security teams operate, situate, and collaborate with partner teams in their organization.

Toward a Field Study on the Impact of Hacking Competitions on Secure Development

The initial results indicate that CTFs have a positive effect on security thinking, encourage communication with the security team, and reduce overconfidence in participants’ ability handle complex security problems.

"You've Got Your Nice List of Bugs, Now What?" Vulnerability Discovery and Management Processes in the Wild

It is found that organizations often struggle with vulnerability remediation and that vulnerability discovery efforts are hindered by significant trust, communication, funding, and staffing issues.

Bug Bounty Marketplaces and Enabling Responsible Vulnerability Disclosure: An Empirical Analysis

The research findings suggest that BBPs are valuable opportunities to source vulnerabilities in software; nevertheless, the rate of disclosure and hacker participation marginally increases with vendor's rewards and other incentives.

A longitudinal study of hacker behaviour

It is concluded that a significant number of inactive and unproductive hackers may contribute, in part, to the difficulties faced by programme operators.

A Survey on Ethical Hacking: Issues and Challenges

The main focus of this paper is to explain the technical and non-technical steps of penetration tests, to make existing systems and their corresponding data more secure, efficient and resilient.

Vulnerability Detection is Just the Beginning

  • Sarah Elder
  • Computer Science
    2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)
  • 2021
This research examines the relationships between the vulnerability detection technique used to find a vulnerability, the type of vulnerability found, the exploitability of the vulnerability, and the effort needed to fix a vulnerability on two projects where all vulnerabilities found have been fixed.



Game of detections: how are security vulnerabilities discovered in the wild?

This first study on vulnerability repositories that targets the reporters of the most common security vulnerabilities, thus concentrating on the people involved in the process, finds that the communities differ based on the security vulnerability they target; but within a specific community, reporters follow similar approaches.

Questions developers ask while diagnosing potential security vulnerabilities with static analysis

An exploratory study with novice and experienced software developers equipped with Find Security Bugs and observed their interactions with security vulnerabilities in an open-source system that they had previously contributed to found that they asked questions not only about security vulnerabilities, associated attacks, and fixes, but also questions about the software itself, the social ecosystem that built the software, and related resources and tools.

Discovering buffer overflow vulnerabilities in the wild: an empirical study

An empirical study on reporters of buffer overflow vulnerabilities to understand the methods and tools used during the discovery found that in spite of many apparent choices, reporters follow similar approaches.

Crowdsourced Security Vulnerability Discovery: Modeling and Organizing Bug-Bounty Programs

An economic model for bug bounties is built and a new policy is proposed that aims to efficiently allocate valuable but scarce hacker effort over time, and across organizations with different crowdsourcing requirements.

An Empirical Study on the Effectiveness of Security Code Review

None of the subjects found all confirmed vulnerabilities, more experience does not necessarily mean that the reviewer will be more accurate or effective, and reports of false vulnerabilities were significantly correlated with reports of valid vulnerabilities.

Improving software security with static automated code analysis in an industry setting

A case study was conducted to evaluate static code analysis in industry focusing on defect detection capability, deployment, and usage of static automated code analysis with a focus on software security, finding that the tool was capable of detecting memory related vulnerabilities, but few vulnerabilities of other types.

Comparing the Effectiveness of Penetration Testing and Static Code Analysis on the Detection of SQL Injection Vulnerabilities in Web Services

  • Nuno AntunesM. Vieira
  • Computer Science
    2009 15th IEEE Pacific Rim International Symposium on Dependable Computing
  • 2009
This work used several commercial and open source tools to detect vulnerabilities in a set of vulnerable services and suggested that, in general, static code analyzers are able to detect more SQL Injection vulnerabilities than penetration testing tools.

Market for Software Vulnerabilities? Think Again

It is demonstrated that an active unregulated market-based mechanism for vulnerabilities almost always underperforms a passive CERT-type mechanism, and it is extended to show that a proposed mechanism--federally funded social planner--always performs better than a market- based mechanism.

One Technique is Not Enough: A Comparison of Vulnerability Discovery Techniques

  • Andrew AustinL. Williams
  • Computer Science
    2011 International Symposium on Empirical Software Engineering and Measurement
  • 2011
The most effective vulnerability discovery technique in terms of vulnerabilities discovered per hour was automated penetration testing, and the results suggest that if one has limited time to preform vulnerability discovery one should conduct automated penetrationTesting to discover implementation bugs and systematic manual penetration testing to discover design flaws.

You Get Where You're Looking for: The Impact of Information Sources on Code Security

Analyzing how the use of information resources impacts code security confirms that API documentation is secure but hard to use, while informal documentation such as Stack Overflow is more accessible but often leads to insecurity.