Guidelines for using financial incentives in software-engineering experimentation

Kruger J, Calikli G, Bershadskyy D, Otto S, Zabel S, Heyer R (2024)
Empirical Software Engineering 29(5): 135.

Zeitschriftenaufsatz | Veröffentlicht | Englisch
 
Download
Es wurden keine Dateien hochgeladen. Nur Publikationsnachweis!
Autor*in
Kruger, Jacob; Calikli, Gul; Bershadskyy, Dmitri; Otto, Siegmar; Zabel, Sarah; Heyer, RobertUniBi
Abstract / Bemerkung
Context:Empirical studies with human participants (e.g., controlled experiments) are established methods in Software Engineering (SE) research to understand developers' activities or the pros and cons of a technique, tool, or practice. Various guidelines and recommendations on designing and conducting different types of empirical studies in SE exist. However, the use of financial incentives (i.e., paying participants to compensate for their effort and improve the validity of a study) is rarely mentionedObjective:In this article, we analyze and discuss the use of financial incentives for human-oriented SE experimentation to derive corresponding guidelines and recommendations for researchers. Specifically, we propose how to extend the current state-of-the-art and provide a better understanding of when and how to incentivize.Method:We captured the state-of-the-art in SE by performing a Systematic Literature Review (SLR) involving 105 publications from six conferences and five journals published in 2020 and 2021. Then, we conducted an interdisciplinary analysis based on guidelines from experimental economics and behavioral psychology, two disciplines that research and use financial incentives.Results:Our results show that financial incentives are sparsely used in SE experimentation, mostly as completion fees. Especially performance-based and task-related financial incentives (i.e., payoff functions) are not used, even though we identified studies for which the validity may benefit from tailored payoff functions. To tackle this issue, we contribute an overview of how experiments in SE may benefit from financial incentivisation, a guideline for deciding on their use, and 11 recommendations on how to design them.Conclusions:We hope that our contributions get incorporated into standards (e.g., the ACM SIGSOFT Empirical Standards), helping researchers understand whether the use of financial incentives is useful for their experiments and how to define a suitable incentivisation strategy.
Stichworte
Empirical software engineering; Experimentation; Financial incentives; Study design; Guidelines
Erscheinungsjahr
2024
Zeitschriftentitel
Empirical Software Engineering
Band
29
Ausgabe
5
Art.-Nr.
135
ISSN
1382-3256
eISSN
1573-7616
Page URI
https://pub.uni-bielefeld.de/record/2991967

Zitieren

Kruger J, Calikli G, Bershadskyy D, Otto S, Zabel S, Heyer R. Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering . 2024;29(5): 135.
Kruger, J., Calikli, G., Bershadskyy, D., Otto, S., Zabel, S., & Heyer, R. (2024). Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering , 29(5), 135. https://doi.org/10.1007/s10664-024-10517-w
Kruger, Jacob, Calikli, Gul, Bershadskyy, Dmitri, Otto, Siegmar, Zabel, Sarah, and Heyer, Robert. 2024. “Guidelines for using financial incentives in software-engineering experimentation”. Empirical Software Engineering 29 (5): 135.
Kruger, J., Calikli, G., Bershadskyy, D., Otto, S., Zabel, S., and Heyer, R. (2024). Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering 29:135.
Kruger, J., et al., 2024. Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering , 29(5): 135.
J. Kruger, et al., “Guidelines for using financial incentives in software-engineering experimentation”, Empirical Software Engineering , vol. 29, 2024, : 135.
Kruger, J., Calikli, G., Bershadskyy, D., Otto, S., Zabel, S., Heyer, R.: Guidelines for using financial incentives in software-engineering experimentation. Empirical Software Engineering . 29, : 135 (2024).
Kruger, Jacob, Calikli, Gul, Bershadskyy, Dmitri, Otto, Siegmar, Zabel, Sarah, and Heyer, Robert. “Guidelines for using financial incentives in software-engineering experimentation”. Empirical Software Engineering 29.5 (2024): 135.
Export

Markieren/ Markierung löschen
Markierte Publikationen

Open Data PUB

Web of Science

Dieser Datensatz im Web of Science®
Suchen in

Google Scholar