Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
Gould E, Fraser HS, Parker TH, Nakagawa S, Griffith SC, Vesk PA, Fidler F, Hamilton DG, Abbey-Lee RN, Abbott JK, Aguirre LA, et al. (2025)
BMC Biology 23(1): 35.
Zeitschriftenaufsatz
| Veröffentlicht | Englisch
Download

Autor*in
Gould, Elliot;
Fraser, Hannah S.;
Parker, Timothy H.;
Nakagawa, Shinichi;
Griffith, Simon C.;
Vesk, Peter A.;
Fidler, Fiona;
Hamilton, Daniel G.;
Abbey-Lee, Robin N.;
Abbott, Jessica K.;
Aguirre, Luis A.;
Alcaraz, Carles
Alle
Alle
Einrichtung
Abstract / Bemerkung
Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.
Stichworte
Analytical heterogeneity;
Metascience;
Many-analyst;
Replication crisis;
Reproducibility
Erscheinungsjahr
2025
Zeitschriftentitel
BMC Biology
Band
23
Ausgabe
1
Art.-Nr.
35
Urheberrecht / Lizenzen
eISSN
1741-7007
Page URI
https://pub.uni-bielefeld.de/record/3000682
Zitieren
Gould E, Fraser HS, Parker TH, et al. Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biology. 2025;23(1): 35.
Gould, E., Fraser, H. S., Parker, T. H., Nakagawa, S., Griffith, S. C., Vesk, P. A., Fidler, F., et al. (2025). Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biology, 23(1), 35. https://doi.org/10.1186/s12915-024-02101-x
Gould, Elliot, Fraser, Hannah S., Parker, Timothy H., Nakagawa, Shinichi, Griffith, Simon C., Vesk, Peter A., Fidler, Fiona, et al. 2025. “Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology”. BMC Biology 23 (1): 35.
Gould, E., Fraser, H. S., Parker, T. H., Nakagawa, S., Griffith, S. C., Vesk, P. A., Fidler, F., Hamilton, D. G., Abbey-Lee, R. N., Abbott, J. K., et al. (2025). Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biology 23:35.
Gould, E., et al., 2025. Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biology, 23(1): 35.
E. Gould, et al., “Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology”, BMC Biology, vol. 23, 2025, : 35.
Gould, E., Fraser, H.S., Parker, T.H., Nakagawa, S., Griffith, S.C., Vesk, P.A., Fidler, F., Hamilton, D.G., Abbey-Lee, R.N., Abbott, J.K., Aguirre, L.A., Alcaraz, C., Aloni, I., Altschul, D., Arekar, K., Atkins, J.W., Atkinson, J., Baker, C.M., Barrett, M., Bell, K., Bello, S.K., Beltrán, I., Berauer, B.J., Bertram, M.G., Billman, P.D., Blake, C.K., Blake, S., Bliard, L., Bonisoli-Alquati, A., Bonnet, T., Bordes, C.N.M., Bose, A.P.H., Botterill-James, T., Boyd, M.A., Boyle, S.A., Bradfer-Lawrence, T., Bradham, J., Brand, J.A., Brengdahl, M.I., Bulla, M., Bussière, L., Camerlenghi, E., Campbell, S.E., Campos, L.L.F., Caravaggi, A., Cardoso, P., Carroll, C.J.W., Catanach, T.A., Chen, X., Chik, H.Y.J., Choy, E.S., Christie, A.P., Chuang, A., Chunco, A.J., Clark, B.L., Contina, A., Covernton, G.A., Cox, M.P., Cressman, K.A., Crotti, M., Crouch, C.D., D’Amelio, P.B., de Sousa, A.A., Döbert, T.F., Dobler, R., Dobson, A.J., Doherty, T.S., Drobniak, S.M., Duffy, A.G., Duncan, A.B., Dunn, R.P., Dunning, J., Dutta, T., Eberhart-Hertel, L., Elmore, J.A., Elsherif, M.M., English, H.M., Ensminger, D.C., Ernst, U.R., Ferguson, S.M., Fernandez-Juricic, E., Ferreira-Arruda, T., Fieberg, J., Finch, E.A., Fiorenza, E.A., Fisher, D.N., Fontaine, A., Forstmeier, W., Fourcade, Y., Frank, G.S., Freund, C.A., Fuentes-Lillo, E., Gandy, S.L., Gannon, D.G., García-Cervigón, A.I., Garretson, A.C., Ge, X., Geary, W.L., Géron, C., Gilles, M., Girndt, A., Gliksman, D., Goldspiel, H.B., Gomes, D.G.E., Good, M.K., Goslee, S.C., Gosnell, J.S., Grames, E.M., Gratton, P., Grebe, N.M., Greenler, S.M., Griffioen, M., Griffith, D.M., Griffith, F.J., Grossman, J.J., Güncan, A., Haesen, S., Hagan, J.G., Hager, H.A., Harris, J.P., Harrison, N.D., Hasnain, S.S., Havird, J.C., Heaton, A.J., Herrera-Chaustre, M.L., Howard, T.J., Hsu, B.-Y., Iannarilli, F., Iranzo, E.C., Iverson, E.N.K., Jimoh, S.O., Johnson, D.H., Johnsson, M., Jorna, J., Jucker, T., Jung, M., Kačergytė, I., Kaltz, O., Ke, A., Kelly, C.D., Keogan, K., Keppeler, F.W., Killion, A.K., Kim, D., Kochan, D.P., Korsten, P., Kothari, S., Kuppler, J., Kusch, J.M., Lagisz, M., Lalla, K.M., Larkin, D.J., Larson, C.L., Lauck, K.S., Lauterbur, M.E., Law, A., Léandri-Breton, D.-J., Lembrechts, J.J., L’Herpiniere, K., Lievens, E.J.P., de Lima, D.O., Lindsay, S., Luquet, M., MacLeod, R., Macphie, K.H., Magellan, K., Mair, M.M., Malm, L.E., Mammola, S., Mandeville, C.P., Manhart, M., Manrique-Garzon, L.M., Mäntylä, E., Marchand, P., Marshall, B.M., Martin, C.A., Martin, D.A., Martin, J.M., Martinig, A.R., McCallum, E.S., McCauley, M., McNew, S.M., Meiners, S.J., Merkling, T., Michelangeli, M., Moiron Cacharron, M., Moreira, B., Mortensen, J., Mos, B., Muraina, T.O., Murphy, P.W., Nelli, L., Niemelä, P., Nightingale, J., Nilsonne, G., Nolazco, S., Nooten, S.S., Novotny, J.L., Olin, A.B., Organ, C.L., Ostevik, K.L., Palacio, F.X., Paquet, M., Parker, D.J., Pascall, D.J., Pasquarella, V.J., Paterson, J.H., Payo-Payo, A., Pedersen, K.M., Perez, G., Perry, K.I., Pottier, P., Proulx, M.J., Proulx, R., Pruett, J.L., Ramananjato, V., Randimbiarison, F.T., Razafindratsima, O.H., Rennison, D.J., Riva, F., Riyahi, S., Roast, M.J., Rocha, F.P., Roche, D.G., Román-Palacios, C., Rosenberg, M.S., Ross, J., Rowland, F.E., Rugemalila, D., Russell, A.L., Ruuskanen, S., Saccone, P., Sadeh, A., Salazar, S.M., Sales, K., Salmón, P., Sanchez-Tojar, A., Santos, L.P., Santostefano, F., Schilling, H.T., Schmidt, M., Schmoll, T., Schneider, A.C., Schrock, A.E., Schroeder, J., Schtickzelle, N., Schultz, N.L., Scott, D.A., Scroggie, M.P., Shapiro, J.T., Sharma, N., Shearer, C.L., Simón, D., Sitvarin, M.I., Skupien, F.L., Slinn, H.L., Smith, G.P., Smith, J.A., Sollmann, R., Whitney, K.S., Still, S.M., Stuber, E.F., Sutton, G.F., Swallow, B., Taff, C.C., Takola, E., Tanentzap, A.J., Tarjuelo, R., Telford, R.J., Thawley, C.J., Thierry, H., Thomson, J., Tidau, S., Tompkins, E.M., Tortorelli, C.M., Trlica, A., Turnell, B.R., Urban, L., Van de Vondel, S., van der Wal, J.E.M., Van Eeckhoven, J., van Oordt, F., Vanderwel, K.M., Vanderwel, M.C., Vanderwolf, K.J., Vélez, J., Vergara-Florez, D.C., Verrelli, B.C., Vieira, M.V., Villamil, N., Vitali, V., Vollering, J., Walker, J., Walker, X.J., Walter, J.A., Waryszak, P., Weaver, R.J., Wedegärtner, R.E.M., Weller, D.L., Whelan, S., White, R.L., Wolfson, D.W., Wood, A., Yanco, S.W., Yen, J.D.L., Youngflesh, C., Zilio, G., Zimmer, C., Zimmerman, G.M., Zitomer, R.A.: Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. BMC Biology. 23, : 35 (2025).
Gould, Elliot, Fraser, Hannah S., Parker, Timothy H., Nakagawa, Shinichi, Griffith, Simon C., Vesk, Peter A., Fidler, Fiona, Hamilton, Daniel G., Abbey-Lee, Robin N., Abbott, Jessica K., Aguirre, Luis A., Alcaraz, Carles, Aloni, Irith, Altschul, Drew, Arekar, Kunal, Atkins, Jeff W., Atkinson, Joe, Baker, Christopher M., Barrett, Meghan, Bell, Kristian, Bello, Suleiman Kehinde, Beltrán, Iván, Berauer, Bernd J., Bertram, Michael Grant, Billman, Peter D., Blake, Charlie K., Blake, Shannon, Bliard, Louis, Bonisoli-Alquati, Andrea, Bonnet, Timothée, Bordes, Camille Nina Marion, Bose, Aneesh P. H., Botterill-James, Thomas, Boyd, Melissa Anna, Boyle, Sarah A., Bradfer-Lawrence, Tom, Bradham, Jennifer, Brand, Jack A., Brengdahl, Martin I., Bulla, Martin, Bussière, Luc, Camerlenghi, Ettore, Campbell, Sara E., Campos, Leonardo L. F., Caravaggi, Anthony, Cardoso, Pedro, Carroll, Charles J. W., Catanach, Therese A., Chen, Xuan, Chik, Heung Ying Janet, Choy, Emily Sarah, Christie, Alec Philip, Chuang, Angela, Chunco, Amanda J., Clark, Bethany L., Contina, Andrea, Covernton, Garth A., Cox, Murray P., Cressman, Kimberly A., Crotti, Marco, Crouch, Connor Davidson, D’Amelio, Pietro B., de Sousa, Alexandra Allison, Döbert, Timm Fabian, Dobler, Ralph, Dobson, Adam J., Doherty, Tim S., Drobniak, Szymon Marian, Duffy, Alexandra Grace, Duncan, Alison B., Dunn, Robert P., Dunning, Jamie, Dutta, Trishna, Eberhart-Hertel, Luke, Elmore, Jared Alan, Elsherif, Mahmoud Medhat, English, Holly M., Ensminger, David C., Ernst, Ulrich Rainer, Ferguson, Stephen M., Fernandez-Juricic, Esteban, Ferreira-Arruda, Thalita, Fieberg, John, Finch, Elizabeth A., Fiorenza, Evan A., Fisher, David N., Fontaine, Amélie, Forstmeier, Wolfgang, Fourcade, Yoan, Frank, Graham S., Freund, Cathryn A., Fuentes-Lillo, Eduardo, Gandy, Sara L., Gannon, Dustin G., García-Cervigón, Ana I., Garretson, Alexis C., Ge, Xuezhen, Geary, William L., Géron, Charly, Gilles, Marc, Girndt, Antje, Gliksman, Daniel, Goldspiel, Harrison B., Gomes, Dylan G. E., Good, Megan Kate, Goslee, Sarah C., Gosnell, J. Stephen, Grames, Eliza M., Gratton, Paolo, Grebe, Nicholas M., Greenler, Skye M., Griffioen, Maaike, Griffith, Daniel M., Griffith, Frances J., Grossman, Jake J., Güncan, Ali, Haesen, Stef, Hagan, James G., Hager, Heather A., Harris, Jonathan Philo, Harrison, Natasha Dean, Hasnain, Sarah Syedia, Havird, Justin Chase, Heaton, Andrew J., Herrera-Chaustre, María Laura, Howard, Tanner J., Hsu, Bin-Yan, Iannarilli, Fabiola, Iranzo, Esperanza C., Iverson, Erik N. K., Jimoh, Saheed Olaide, Johnson, Douglas H., Johnsson, Martin, Jorna, Jesse, Jucker, Tommaso, Jung, Martin, Kačergytė, Ineta, Kaltz, Oliver, Ke, Alison, Kelly, Clint D., Keogan, Katharine, Keppeler, Friedrich Wolfgang, Killion, Alexander K., Kim, Dongmin, Kochan, David P., Korsten, Peter, Kothari, Shan, Kuppler, Jonas, Kusch, Jillian M., Lagisz, Malgorzata, Lalla, Kristen Marianne, Larkin, Daniel J., Larson, Courtney L., Lauck, Katherine S., Lauterbur, M. Elise, Law, Alan, Léandri-Breton, Don-Jean, Lembrechts, Jonas J., L’Herpiniere, Kiara, Lievens, Eva J. P., de Lima, Daniela Oliveira, Lindsay, Shane, Luquet, Martin, MacLeod, Ross, Macphie, Kirsty H., Magellan, Kit, Mair, Magdalena M., Malm, Lisa E., Mammola, Stefano, Mandeville, Caitlin P., Manhart, Michael, Manrique-Garzon, Laura Milena, Mäntylä, Elina, Marchand, Philippe, Marshall, Benjamin Michael, Martin, Charles A., Martin, Dominic Andreas, Martin, Jake Mitchell, Martinig, April Robin, McCallum, Erin S., McCauley, Mark, McNew, Sabrina M., Meiners, Scott J., Merkling, Thomas, Michelangeli, Marcus, Moiron Cacharron, Maria, Moreira, Bruno, Mortensen, Jennifer, Mos, Benjamin, Muraina, Taofeek Olatunbosun, Murphy, Penelope Wrenn, Nelli, Luca, Niemelä, Petri, Nightingale, Josh, Nilsonne, Gustav, Nolazco, Sergio, Nooten, Sabine S., Novotny, Jessie Lanterman, Olin, Agnes Birgitta, Organ, Chris L., Ostevik, Kate L., Palacio, Facundo Xavier, Paquet, Matthieu, Parker, Darren James, Pascall, David J., Pasquarella, Valerie J., Paterson, John Harold, Payo-Payo, Ana, Pedersen, Karen Marie, Perez, Grégoire, Perry, Kayla I., Pottier, Patrice, Proulx, Michael J., Proulx, Raphaël, Pruett, Jessica L, Ramananjato, Veronarindra, Randimbiarison, Finaritra Tolotra, Razafindratsima, Onja H., Rennison, Diana J., Riva, Federico, Riyahi, Sepand, Roast, Michael James, Rocha, Felipe Pereira, Roche, Dominique G., Román-Palacios, Cristian, Rosenberg, Michael S., Ross, Jessica, Rowland, Freya E., Rugemalila, Deusdedith, Russell, Avery L., Ruuskanen, Suvi, Saccone, Patrick, Sadeh, Asaf, Salazar, Stephen M., Sales, Kris, Salmón, Pablo, Sanchez-Tojar, Alfredo, Santos, Leticia Pereira, Santostefano, Francesca, Schilling, Hayden T., Schmidt, Marcus, Schmoll, Tim, Schneider, Adam C., Schrock, Allie E., Schroeder, Julia, Schtickzelle, Nicolas, Schultz, Nick L., Scott, Drew A., Scroggie, Michael Peter, Shapiro, Julie Teresa, Sharma, Nitika, Shearer, Caroline L., Simón, Diego, Sitvarin, Michael I., Skupien, Fabrício Luiz, Slinn, Heather Lea, Smith, Grania Polly, Smith, Jeremy A., Sollmann, Rahel, Whitney, Kaitlin Stack, Still, Shannon Michael, Stuber, Erica F., Sutton, Guy F., Swallow, Ben, Taff, Conor Claverie, Takola, Elina, Tanentzap, Andrew J., Tarjuelo, Rocío, Telford, Richard J., Thawley, Christopher J., Thierry, Hugo, Thomson, Jacqueline, Tidau, Svenja, Tompkins, Emily M., Tortorelli, Claire Marie, Trlica, Andrew, Turnell, Biz R., Urban, Lara, Van de Vondel, Stijn, van der Wal, Jessica Eva Megan, Van Eeckhoven, Jens, van Oordt, Francis, Vanderwel, K. Michelle, Vanderwel, Mark C., Vanderwolf, Karen J., Vélez, Juliana, Vergara-Florez, Diana Carolina, Verrelli, Brian C., Vieira, Marcus Vinícius, Villamil, Nora, Vitali, Valerio, Vollering, Julien, Walker, Jeffrey, Walker, Xanthe J., Walter, Jonathan A., Waryszak, Pawel, Weaver, Ryan J., Wedegärtner, Ronja E. M., Weller, Daniel L., Whelan, Shannon, White, Rachel Louise, Wolfson, David William, Wood, Andrew, Yanco, Scott W., Yen, Jian D. L., Youngflesh, Casey, Zilio, Giacomo, Zimmer, Cédric, Zimmerman, Gregory Mark, and Zitomer, Rachel A. “Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology”. BMC Biology 23.1 (2025): 35.
Alle Dateien verfügbar unter der/den folgenden Lizenz(en):
Creative Commons Namensnennung - Nicht kommerziell - Keine Bearbeitungen 4.0 International (CC BY-NC-ND 4.0):
Volltext(e)
Name
s12915-024-02101-x.pdf
2.74 MB
Access Level

Zuletzt Hochgeladen
2025-02-26T14:27:11Z
MD5 Prüfsumme
deb66bd4ab45c992d631cf0594160c4a
Link(s) zu Volltext(en)
Access Level

Daten bereitgestellt von European Bioinformatics Institute (EBI)
Zitationen in Europe PMC
Daten bereitgestellt von Europe PubMed Central.
References
Daten bereitgestellt von Europe PubMed Central.
Export
Markieren/ Markierung löschen
Markierte Publikationen
Web of Science
Dieser Datensatz im Web of Science®Quellen
PMID: 39915771
PubMed | Europe PMC
Suchen in