The reliability of replications: a study in computational reproductions

Breznau N, Rinke EM, Wuttke A, Adem M, Adriaans J, Akdeniz E, Alvarez-Benjumea A, Andersen HK, Auer D, Azevedo F, Bahnsen O, et al. (2025)
Royal Society Open Science 12(3).

Zeitschriftenaufsatz | Veröffentlicht | Englisch
 
Download
OA 1.10 MB
Autor*in
Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander; Adem, Muna; Adriaans, JuleUniBi ; Akdeniz, Esra; Alvarez-Benjumea, Amalia; Andersen, Henrik K.; Auer, Daniel; Azevedo, Flavio; Bahnsen, Oke; Bai, Ling
Alle
Abstract / Bemerkung
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign andp-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.
Erscheinungsjahr
2025
Zeitschriftentitel
Royal Society Open Science
Band
12
Ausgabe
3
eISSN
2054-5703
Page URI
https://pub.uni-bielefeld.de/record/3001790

Zitieren

Breznau N, Rinke EM, Wuttke A, et al. The reliability of replications: a study in computational reproductions. Royal Society Open Science. 2025;12(3).
Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Alvarez-Benjumea, A., et al. (2025). The reliability of replications: a study in computational reproductions. Royal Society Open Science, 12(3). https://doi.org/10.1098/rsos.241038
Breznau, Nate, Rinke, Eike Mark, Wuttke, Alexander, Adem, Muna, Adriaans, Jule, Akdeniz, Esra, Alvarez-Benjumea, Amalia, et al. 2025. “The reliability of replications: a study in computational reproductions”. Royal Society Open Science 12 (3).
Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., et al. (2025). The reliability of replications: a study in computational reproductions. Royal Society Open Science 12.
Breznau, N., et al., 2025. The reliability of replications: a study in computational reproductions. Royal Society Open Science, 12(3).
N. Breznau, et al., “The reliability of replications: a study in computational reproductions”, Royal Society Open Science, vol. 12, 2025.
Breznau, N., Rinke, E.M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Alvarez-Benjumea, A., Andersen, H.K., Auer, D., Azevedo, F., Bahnsen, O., Bai, L., Balzer, D., Bauer, P.C., Bauer, G., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., Berthold, A., Bethke, F.S., Biegert, T., Blinzler, K., Blumenberg, J.N., Bobzien, L., Bohman, A., Bol, T., Bostic, A., Brzozowska, Z., Burgdorf, K., Burger, K., Busch, K., Castillo, J.-C., Chan, N., Christmann, P., Connelly, R., Czymara, C.S., Damian, E., de Rooij, E.A., Ecker, A., Edelmann, A., Eder, C., Eger, M.A., Ellerbrock, S., Forke, A., Forster, A., Freire, D., Gaasendam, C., Gavras, K., Gayle, V., Gessler, T., Gnambs, T., Godefroidt, A., Grömping, M., Groß, M., Gruber, S., Gummer, T., Hadjar, A., Halbherr, V., Heisig, J.P., Hellmeier, S., Heyne, S., Hirsch, M., Hjerm, M., Hochman, O., Höffler, J.H., Hövermann, A., Hunger, S., Hunkler, C., Huth-Stöckle, N., Ignácz, Z.S., Israel, S., Jacobs, L., Jacobsen, J., Jaeger, B., Jungkunz, S., Jungmann, N., Kanjana, J., Kauff, M., Khan, S., Khatua, S., Kleinert, M., Klinger, J., Kolb, J.-P., Kołczyńska, M., Kuk, J., Kunißen, K., Kurti Sinatra, D., Langenkamp, A., Lee, R.C., Lersch, P.M., Liu, D., Löbel, L.-M., Lutscher, P., Mader, M., Madia, J.E., Malancu, N., Maldonado, L., Marahrens, H., Martin, N., Martinez, P., Mayerl, J., Mayorga, O.J., McDonnell, R., McManus, P., McWagner, K., Meeusen, C., Meierrieks, D., Mellon, J., Merhout, F., Merk, S., Meyer, D., Micheli, L., Mijs, J., Moya, C., Neunhoeffer, M., Nüst, D., Nygård, O., Ochsenfeld, F., Otte, G., Pechenkina, A., Pickup, M., Prosser, C., Raes, L., Ralston, K., Ramos, M., Reichert, F., Roets, A., Rogers, J., Ropers, G., Samuel, R., Sand, G., Sanhueza Petrarca, C., Schachter, A., Schaeffer, M., Schieferdecker, D., Schlueter, E., Schmidt, K., Schmidt, R., Schmidt-Catran, A., Schmiedeberg, C., Schneider, J., Schoonvelde, M., Schulte-Cloos, J., Schumann, S., Schunck, R., Seuring, J., Silber, H., Sleegers, W., Sonntag, N., Staudt, A., Steiber, N., Steiner, N.D., Sternberg, S., Stiers, D., Stojmenovska, D., Storz, N., Striessnig, E., Stroppe, A.-K., Suchow, J.W., Teltemann, J., Tibajev, A., Tung, B., Vagni, G., Van Assche, J., van der Linden, M., van der Noll, J., Van Hootegem, A., Vogtenhuber, S., Voicu, B., Wagemans, F., Wehl, N., Werner, H., Wiernik, B.M., Winter, F., Wolf, C., Wu, C., Yamada, Y., Zakula, B., Zhang, N., Ziller, C., Zins, S., Żółtak, T., Nguyen, H.H.V.: The reliability of replications: a study in computational reproductions. Royal Society Open Science. 12, (2025).
Breznau, Nate, Rinke, Eike Mark, Wuttke, Alexander, Adem, Muna, Adriaans, Jule, Akdeniz, Esra, Alvarez-Benjumea, Amalia, Andersen, Henrik K., Auer, Daniel, Azevedo, Flavio, Bahnsen, Oke, Bai, Ling, Balzer, Dave, Bauer, Paul C., Bauer, Gerrit, Baumann, Markus, Baute, Sharon, Benoit, Verena, Bernauer, Julian, Berning, Carl, Berthold, Anna, Bethke, Felix S., Biegert, Thomas, Blinzler, Katharina, Blumenberg, Johannes N., Bobzien, Licia, Bohman, Andrea, Bol, Thijs, Bostic, Amie, Brzozowska, Zuzanna, Burgdorf, Katharina, Burger, Kaspar, Busch, Kathrin, Castillo, Juan-Carlos, Chan, Nathan, Christmann, Pablo, Connelly, Roxanne, Czymara, Christian S., Damian, Elena, de Rooij, Eline A., Ecker, Alejandro, Edelmann, Achim, Eder, Christina, Eger, Maureen A., Ellerbrock, Simon, Forke, Anna, Forster, Andrea, Freire, Danilo, Gaasendam, Chris, Gavras, Konstantin, Gayle, Vernon, Gessler, Theresa, Gnambs, Timo, Godefroidt, Amélie, Grömping, Max, Groß, Martin, Gruber, Stefan, Gummer, Tobias, Hadjar, Andreas, Halbherr, Verena, Heisig, Jan Paul, Hellmeier, Sebastian, Heyne, Stefanie, Hirsch, Magdalena, Hjerm, Mikael, Hochman, Oshrat, Höffler, Jan H., Hövermann, Andreas, Hunger, Sophia, Hunkler, Christian, Huth-Stöckle, Nora, Ignácz, Zsófia S., Israel, Sabine, Jacobs, Laura, Jacobsen, Jannes, Jaeger, Bastian, Jungkunz, Sebastian, Jungmann, Nils, Kanjana, Jennifer, Kauff, Mathias, Khan, Salman, Khatua, Sayak, Kleinert, Manuel, Klinger, Julia, Kolb, Jan-Philipp, Kołczyńska, Marta, Kuk, John, Kunißen, Katharina, Kurti Sinatra, Dafina, Langenkamp, Alexander, Lee, Robin C., Lersch, Philipp M., Liu, David, Löbel, Lea-Maria, Lutscher, Philipp, Mader, Matthias, Madia, Joan E., Malancu, Natalia, Maldonado, Luis, Marahrens, Helge, Martin, Nicole, Martinez, Paul, Mayerl, Jochen, Mayorga, Oscar J., McDonnell, Robert, McManus, Patricia, McWagner, Kyle, Meeusen, Cecil, Meierrieks, Daniel, Mellon, Jonathan, Merhout, Friedolin, Merk, Samuel, Meyer, Daniel, Micheli, Leticia, Mijs, Jonathan, Moya, Cristóbal, Neunhoeffer, Marcel, Nüst, Daniel, Nygård, Olav, Ochsenfeld, Fabian, Otte, Gunnar, Pechenkina, Anna, Pickup, Mark, Prosser, Christopher, Raes, Louis, Ralston, Kevin, Ramos, Miguel, Reichert, Frank, Roets, Arne, Rogers, Jonathan, Ropers, Guido, Samuel, Robin, Sand, Gregor, Sanhueza Petrarca, Constanza, Schachter, Ariela, Schaeffer, Merlin, Schieferdecker, David, Schlueter, Elmar, Schmidt, Katja, Schmidt, Regine, Schmidt-Catran, Alexander, Schmiedeberg, Claudia, Schneider, Jürgen, Schoonvelde, Martijn, Schulte-Cloos, Julia, Schumann, Sandy, Schunck, Reinhard, Seuring, Julian, Silber, Henning, Sleegers, Willem, Sonntag, Nico, Staudt, Alexander, Steiber, Nadia, Steiner, Nils D., Sternberg, Sebastian, Stiers, Dieter, Stojmenovska, Dragana, Storz, Nora, Striessnig, Erich, Stroppe, Anne-Kathrin, Suchow, Jordan W., Teltemann, Janna, Tibajev, Andrey, Tung, Brian, Vagni, Giacomo, Van Assche, Jasper, van der Linden, Meta, van der Noll, Jolanda, Van Hootegem, Arno, Vogtenhuber, Stefan, Voicu, Bogdan, Wagemans, Fieke, Wehl, Nadja, Werner, Hannah, Wiernik, Brenton M., Winter, Fabian, Wolf, Christof, Wu, Cary, Yamada, Yuki, Zakula, Björn, Zhang, Nan, Ziller, Conrad, Zins, Stefan, Żółtak, Tomasz, and Nguyen, Hung H.V. “The reliability of replications: a study in computational reproductions”. Royal Society Open Science 12.3 (2025).
Alle Dateien verfügbar unter der/den folgenden Lizenz(en):
Creative Commons Namensnennung 4.0 International Public License (CC-BY 4.0):
Volltext(e)
Access Level
OA Open Access
Zuletzt Hochgeladen
2025-03-24T08:46:42Z
MD5 Prüfsumme
3321082bd087830e5158f25d423e802c


Export

Markieren/ Markierung löschen
Markierte Publikationen

Open Data PUB

Suchen in

Google Scholar