Views on Internal and External Validity in Empirical Software Engineering

@article{Siegmund2015ViewsOI,
  title={Views on Internal and External Validity in Empirical Software Engineering},
  author={Janet Siegmund and Norbert Siegmund and Sven Apel},
  journal={2015 IEEE/ACM 37th IEEE International Conference on Software Engineering},
  year={2015},
  volume={1},
  pages={9-19}
}
Empirical methods have grown common in software engineering, but there is no consensus on how to apply them properly. [] Key Result We found that the opinions differ considerably, and that there is no consensus in the community when to focus on internal or external validity and how to conduct and review replications.

Figures and Tables from this paper

How reviewers think about internal and external validity in empirical software engineering
TLDR
Empirical methods have grown common in software engineering, but there is no consensus on how to apply them properly, and existing guidelines are too general to assist in making this decision.
From Craft to Science: The Road Ahead for Empirical Software Engineering Research
TLDR
Why, despite persistent discussions in the SE research community, there are still disagreements about why and how to conduct empirical research is explored and how empirical SE research can progress beyond "one-off" studies and endless "new and exciting" results is explored.
Empirical software engineering experts on the use of students and professionals in experiments
TLDR
Using students as participants remains a valid simplification of reality needed in laboratory contexts but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment.
Empirical Research in Software Architecture: How Far have We Come?
  • M. Galster, Danny Weyns
  • Computer Science
    2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA)
  • 2016
TLDR
The results indicate that software architecture has become a more mature domain with regards to applying empirical research, however, there are issues in research practices that could be improved.
Methodology Matters: Is There a Method ChoiceBias in Software Engineering?
TLDR
This paper considers two years of ICSE proceedings and finds that a majority of studies use computer simulation methods relying on trace measures rather than active human participation, which leads to a low level of control over extraneous factors.
The ABC of Software Engineering Research
TLDR
A taxonomy from the social sciences is adopted, termed here the ABC framework for SE research, which offers a holistic view of eight archetypal research strategies, and six ways in which the framework can advance SE research.
Methodology Matters: How We Study Socio-Technical Aspects in Software Engineering
TLDR
It is proposed that the software engineering community should diversify the use of research strategies so that it may have a deeper understanding of human and social aspects of software development practice, while balancing the design and evaluation of innovations on the technical side.
Using Students as Experimental Subjects in Software Engineering Research - A Review and Discussion of the Evidence
TLDR
Based on an extensive review of published works that have compared students to professionals, it is found that picking on "students" is counterproductive for two main reasons and classifying experimental subjects by their status is merely a proxy for more important and meaningful classifications.
Mitigating Threats to Validity in Empirical Software Engineering: A Traceability Case Study
TLDR
A practical approach for mitigating threats to validity in empirical software engineering using a sequence of software activities is provided using a real-world traceability case study for illustration purposes.
Empirical Research in Software Engineering — A Literature Survey
TLDR
This paper identifies both the overall landscape and detailed implementations of ESE, and investigates frequently applied empirical methods, targeted research purposes, used data sources, and applied data processing approaches and tools in ESE.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 44 REFERENCES
On the success of empirical studies in the international conference on software engineering
Critiques of the quantity and quality of empirical evaluations in software engineering have existed for quite some time. However such critiques are typically not empirically evaluated. This paper
Hints for Reviewing Empirical Work in Software Engineering
  • W. Tichy
  • Economics
    Empirical Software Engineering
  • 2004
TLDR
These hints are meant to help with judging empirical work and reduce some of the angst associated with accepting empirical papers.
Performing systematic literature reviews in software engineering
TLDR
This tutorial is designed to provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews, and to gain the knowledge needed to conduct systematic reviews of their own.
A survey of controlled experiments in software engineering
TLDR
This paper reports upon the present state of how controlled experiments in software engineering are conducted and the extent to which relevant information is reported, and quantitatively characterizes the topics and their subjects, tasks, environments, and environments.
The Future of Empirical Methods in Software Engineering Research
TLDR
Means to meet challenges to meet the vision of empirical research methods for software engineering include increased competence regarding how to apply and combine alternative empirical methods, tighter links between academia and industry, the development of common research agendas with a focus on empirical Methods, and more resources for empirical research.
The role of replications in Empirical Software Engineering
TLDR
This paper identifies two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replication, inWhich the same research question is evaluated by using a different experimental procedure.
Reporting guidelines for controlled experiments in software engineering
TLDR
The objective of this paper is to survey the most prominent published proposals for reporting guidelines, and to derive a unified standard that which can serve as a starting point for further discussion and to adapt guideline development processes from other disciplines.
Reporting Experiments in Software Engineering
TLDR
A checklist for reporting results from controlled experiments is expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in finding the information they are looking for.
On the difficulty of replicating human subjects studies in software engineering
TLDR
It is suggested that, for some human subjects studies, literal replication may not be the the most effective strategy for validating the results of previous studies.
Research toward an engineering discipline for software
  • M. Shaw
  • Computer Science
    FoSER '10
  • 2010
TLDR
To establish a true engineering discipline for software, the view of what constitutes a "software system" needs to be broadened and techniques that help to provide cost-effective quality despite associated uncertainties need to be developed.
...
1
2
3
4
5
...