Considerations and Pitfalls in Controlled Experiments on Code Comprehension

  title={Considerations and Pitfalls in Controlled Experiments on Code Comprehension},
  author={Dror G. Feitelson},
  journal={2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)},
  • D. Feitelson
  • Published 15 March 2021
  • Psychology
  • 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)
Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance… 
Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension
  • D. Feitelson
  • Computer Science
    Empirical Software Engineering
  • 2022
In order to promote the development and use of sound experimental methodology, both considerations which need to be applied and potential problems that might occur are discussed, with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.
40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study
A systematicmapping study of 95 source code comprehension experiments published between 1979 and 2019 provides a basis for subsequent discussion of the huge diversity of design options in the face of a lack of basic research on their consequences and comparability.
Anchoring Code Understandability Evaluations Through Task Descriptions
This study investigated in a randomized, controlled experiment whether a hint about the difficulty of the code to be understood in a task description anchors participants in their own code comprehensibility ratings and shows that participants are significantly influenced by the initial scenario description in their assessment of code com-prehensibility.
Causette: User-Controlled Rearrangement of Causal Constructs in a Code Editor
Cusette, a set of four novel interaction techniques for a code editor that consist in rearranging causal constructs on demand to make the code representation consistent with the causal chain being ana-lyzed by the user, is designed.
The Effect of Information Content and Length on Name Recollection
It is shown in two independent experiments that informative names are usually easier to recollect than similar-length names which contain less focused information, and that less-experienced and female participants are better at remembering the less informative names.


Towards standard for experiments in program comprehension
  • V. RajlichG. S. Cowan
  • Computer Science
    Proceedings Fifth International Workshop on Program Comprehension. IWPC'97
  • 1997
It is proposed that program comprehension experiments always measure and interpret the following dependent variables: accuracy, accurate response time, and inaccurate response time.
Confounding parameters on program comprehension: a literature survey
A catalog of 39 confounding parameters is created, including an overview of measurement and control techniques, to support researchers in identifying relevant confounding parameters and select appropriate techniques to control their influence.
On the role of hypotheses during opportunistic understanding while porting large scale code
A case study observing an experienced software engineer porting a large software system and the role of hypotheses in accomplishing the porting task and the results point to information that could help novices to become experts faster.
Program Comprehension: Past, Present, and Future
  • J. Siegmund
  • Computer Science
    2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)
  • 2016
The past of program-comprehension research is explored, the current state is discussed, and what future research on program comprehension might bring is outlined.
Measuring Computer Program Quality and Comprehension
Understanding misunderstandings in source code
To identify code patterns that may confuse programmers, a preliminary set of `atoms of confusion' from known confusing code is extracted and it is shown that these code patterns can lead to a significantly increased rate of misunderstanding versus equivalent code without the patterns.
Program understanding behavior during adaptation of large scale software
  • A. AndrewsA. M. Vans
  • Computer Science
    Proceedings. 6th International Workshop on Program Comprehension. IWPC'98 (Cat. No.98TB100242)
  • 1998
The paper reports on the general understanding process, the types of actions programmers preferred during the adaptation task, and the level of abstraction, at which they were working.
Studying programmer behavior experimentally: the problems of proper methodology
Three major areas of methodological concern, the selection of subjects, materials, and measures, are reviewed and the first two of these areas continue to present major difficulties for this type of research.
Program understanding behaviour during enhancement of large-scale software
It is found that programmers work predominantly at the code and algorithmic levels with differences depending on the stage of the enhancement, with strong indication that memory (over)load is an issue.
How do professional developers comprehend software?
An observational study of 28 professional developers from seven companies, investigating how developers comprehend software finds that developers put themselves in the role of end users by inspecting user interfaces and that face-to-face communication is preferred to documentation.