View source for Journal:Judgements of research co-created by generative AI: Experimental evidence

{{Infobox journal article
|name         = 
|image        = 
|alt          = <!-- Alternative text for images -->
|caption      = 
|title_full   = Judgements of research co-created by generative AI: Experimental evidence
|journal      = ''Economics and Business Review''
|authors      = Niszczota, Paweł; Conway, Paul
|affiliations = Poznań University of Economics and Business, University of Southampton
|contact      = Email: pawel dot niszczota at ue dot poznan dot pl
|editors      = 
|pub_year     = 2023
|vol_iss      = '''9'''(2)
|pages        = 101–114
|doi          = [https://doi.org/10.18559/ebr.2023.2.744 10.18559/ebr.2023.2.744]
|issn         = 2450-0097
|license      = [https://creativecommons.org/licenses/by/4.0/ Creative Commons Attribution 4.0 International]
|website      = [https://journals.ue.poznan.pl/ebr/article/view/744 https://journals.ue.poznan.pl/ebr/article/view/744]
|download     = [https://journals.ue.poznan.pl/ebr/article/view/744/569 https://journals.ue.poznan.pl/ebr/article/view/744/569] (PDF)
}}
==Abstract==
The introduction of [[ChatGPT]] has fuelled a public debate on the appropriateness of using generative [[artificial intelligence]] (AI) ([[large language model]]s or LLMs) in work, including a debate on how they might be used (and abused) by researchers. In the current work, we test whether delegating parts of the research process to LLMs leads people to distrust researchers and devalues their scientific work. Participants (''N'' = 402) considered a researcher who delegates elements of the research process to a PhD student or LLM and rated three aspects of such delegation. Firstly, they rated whether it is morally appropriate to do so. Secondly, they judged whether—after deciding to delegate the research process—they would trust the scientist (who decided to delegate) to oversee future projects. Thirdly, they rated the expected accuracy and [[Quality (business)|quality]] of the output from the delegated research process. Our results show that people judged delegating to an LLM as less morally acceptable than delegating to a human (''d'' = –0.78). Delegation to an LLM also decreased trust to oversee future research projects (''d'' = –0.80), and people thought the results would be less accurate and of lower quality (''d'' = –0.85). We discuss how this devaluation might transfer into the underreporting of generative AI use.

'''Keywords''': trust in science, metascience, ChatGPT, GPT, large language models, generative AI, experiment

==Introduction==
The introduction of [[ChatGPT]] appears to have become a tipping point for [[large language model]]s (LLMs). It is expected that LLMs—such as those released by OpenAI (i.e., ChatGPT and GPT-4)<ref>{{Cite web |last=Schulman, J.; Zoph, B.; Kim, C. et al. |date=30 November 2022 |title=Introducing ChatGPT |url=https://openai.com/blog/chatgpt |publisher=OpenAI}}</ref><ref>{{Cite journal |last=OpenAI |last2=Achiam |first2=Josh |last3=Adler |first3=Steven |last4=Agarwal |first4=Sandhini |last5=Ahmad |first5=Lama |last6=Akkaya |first6=Ilge |last7=Aleman |first7=Florencia Leoni |last8=Almeida |first8=Diogo |last9=Altenschmidt |first9=Janko |last10=Altman |first10=Sam |last11=Anadkat |first11=Shyamal |date=2023-12-18 |title=GPT-4 Technical Report |url=http://arxiv.org/abs/2303.08774 |journal=arXiv:2303.08774 [cs] |doi=10.48550/arxiv.2303.08774}}</ref>, but also major technology firms such as Google and Meta—will impact the work of many white-collar professions.<ref>{{Cite journal |last=Alper |first=Sinan |last2=Yilmaz |first2=Onurcan |date=2020-04 |title=Does an Abstract Mind-Set Increase the Internal Consistency of Moral Attitudes and Strengthen Individualizing Foundations? |url=http://journals.sagepub.com/doi/10.1177/1948550619856309 |journal=Social Psychological and Personality Science |language=en |volume=11 |issue=3 |pages=326–335 |doi=10.1177/1948550619856309 |issn=1948-5506}}</ref><ref>{{Cite journal |last=Eloundou |first=Tyna |last2=Manning |first2=Sam |last3=Mishkin |first3=Pamela |last4=Rock |first4=Daniel |date=2023-08-21 |title=GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models |url=http://arxiv.org/abs/2303.10130 |journal=arXiv:2303.10130 [cs, econ, q-fin] |doi=10.48550/arxiv.2303.10130}}</ref><ref>{{Cite journal |last=Korzynski |first=Pawel |last2=Mazurek |first2=Grzegorz |last3=Altmann |first3=Andreas |last4=Ejdys |first4=Joanna |last5=Kazlauskaite |first5=Ruta |last6=Paliszkiewicz |first6=Joanna |last7=Wach |first7=Krzysztof |last8=Ziemba |first8=Ewa |date=2023-05-30 |title=Generative artificial intelligence as a new context for management theories: analysis of ChatGPT |url=https://www.emerald.com/insight/content/doi/10.1108/CEMJ-02-2023-0091/full/html |journal=Central European Management Journal |language=en |volume=31 |issue=1 |pages=3–13 |doi=10.1108/CEMJ-02-2023-0091 |issn=2658-2430}}</ref> This impact extends to top academic journals such as ''Nature'' and ''Science'', which have already acknowledged the impact [[artificial intelligence]] (AI) has on the scientific profession and started setting out some guides on how to use LLMs.<ref name=":0">{{Cite journal |last=Thorp |first=H. Holden |date=2023-01-27 |title=ChatGPT is fun, but not an author |url=https://www.science.org/doi/10.1126/science.adg7879 |journal=Science |language=en |volume=379 |issue=6630 |pages=313–313 |doi=10.1126/science.adg7879 |issn=0036-8075}}</ref><ref>{{Cite journal |date=2023-01-26 |title=Tools such as ChatGPT threaten transparent science; here are our ground rules for their use |url=https://www.nature.com/articles/d41586-023-00191-1 |journal=Nature |language=en |volume=613 |issue=7945 |pages=612–612 |doi=10.1038/d41586-023-00191-1 |issn=0028-0836}}</ref> For example, listing ChatGPT as a co-author was deemed inappropriate.<ref name=":0" /><ref>{{Cite journal |last=Stokel-Walker |first=Chris |date=2023-01-26 |title=ChatGPT listed as author on research papers: many scientists disapprove |url=https://www.nature.com/articles/d41586-023-00107-z |journal=Nature |language=en |volume=613 |issue=7945 |pages=620–621 |doi=10.1038/d41586-023-00107-z |issn=0028-0836}}</ref> However, the use of such models is not explicitly forbidden; rather, it is suggested that researchers report on which part of the research process they received assistance from ChatGPT.

Important questions remain regarding how scientists employing LLMs in their work are perceived by society.<ref>{{Cite journal |last=Dwivedi |first=Yogesh K. |last2=Kshetri |first2=Nir |last3=Hughes |first3=Laurie |last4=Slade |first4=Emma Louise |last5=Jeyaraj |first5=Anand |last6=Kar |first6=Arpan Kumar |last7=Baabdullah |first7=Abdullah M. |last8=Koohang |first8=Alex |last9=Raghavan |first9=Vishnupriya |last10=Ahuja |first10=Manju |last11=Albanna |first11=Hanaa |date=2023-08 |title=Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy |url=https://linkinghub.elsevier.com/retrieve/pii/S0268401223000233 |journal=International Journal of Information Management |language=en |volume=71 |pages=102642 |doi=10.1016/j.ijinfomgt.2023.102642}}</ref> Do people view the use of LLMs as diminishing the importance, value, and worth of scientific efforts, and if so, which elements of the scientific process does LLM usage most impact? We examine these questions with a study on the perceptions of scientists who rely on an LLM for various aspects of the scientific process.

We anticipated that, overall, people would view the delegation of aspects of the research process to an LLM as morally worse than delegating to a human, and that doing so would reduce trust in the delegating scientist. Moreover, insofar as people view creativity as a core human trait, especially in comparison to AI<ref>{{Cite journal |last=Cha |first=Young-Jae |last2=Baek |first2=Sojung |last3=Ahn |first3=Grace |last4=Lee |first4=Hyoungsuk |last5=Lee |first5=Boyun |last6=Shin |first6=Ji-eun |last7=Jang |first7=Dayk |date=2020-02 |title=Compensating for the loss of human distinctiveness: The use of social creativity under Human–Machine comparisons |url=https://linkinghub.elsevier.com/retrieve/pii/S0747563219303140 |journal=Computers in Human Behavior |language=en |volume=103 |pages=80–90 |doi=10.1016/j.chb.2019.08.027}}</ref>, and some aspects of the research process may entail more creativity than others—such as idea generation and prior literature synthesis<ref>{{Cite journal |last=King |first=Michael |date=2023-04-07 |title=Can GPT-4 formulate and test a novel hypothesis? Yes and no |url=https://www.techrxiv.org/doi/full/10.36227/techrxiv.22517278.v1 |doi=10.36227/techrxiv.22517278.v1}}</ref>, compared to data identification and preparation, testing framework determination and implementation, or results analysis—we tested the exploratory prediction that the effect of delegation to AI versus a human on moral ratings and trust might be different for these aspects.

We contribute to an emerging literature exploring how large language models can assist research on economics and financial economics. The reader can find a valuable discussion on the use of LLMs in economic research in Korinek<ref>{{Cite journal |last=Korinek |first=Anton |date=2023-02 |title=Language Models and Cognitive Automation for Economic Research |url=http://www.nber.org/papers/w30957.pdf |language=en |place=Cambridge, MA |pages=w30957 |doi=10.3386/w30957}}</ref> and Wach ''et al.''<ref>{{Cite journal |last=Wach |first=Krzysztof |last2=Duong |first2=Cong Doanh |last3=Ejdys |first3=Joanna |last4=Kazlauskaitė |first4=Rūta |last5=Korzynski |first5=Pawel |last6=Mazurek |first6=Grzegorz |last7=Paliszkiewicz |first7=Joanna |last8=Ziemba |first8=Ewa |date=2023 |title=The dark side of generative artificial intelligence: A critical analysis of controversies and risks of ChatGPT |url=https://eber.uek.krakow.pl/index.php/eber/article/view/2113 |journal=Entrepreneurial Business and Economics Review |volume=11 |issue=2 |pages=7–30 |doi=10.15678/EBER.2023.110201}}</ref>  A noteworthy empirical study can be found in Dowling and Lucey<ref name=":1">{{Cite journal |last=Dowling |first=Michael |last2=Lucey |first2=Brian |date=2023-05 |title=ChatGPT for (Finance) research: The Bananarama Conjecture |url=https://linkinghub.elsevier.com/retrieve/pii/S1544612323000363 |journal=Finance Research Letters |language=en |volume=53 |pages=103662 |doi=10.1016/j.frl.2023.103662}}</ref>, who asked financial academics to rate research ideas on cryptocurrency, and they judged that the output is of fair [[Quality (business)|quality]].

==Research questions==
We ask two research questions concerning laypeople’s perception of the use of LLMs in science. First, we tested the hypothesis that people will perceive research assistance from LLMs less favorably than the very same assistance from a junior human researcher. In both cases, we assume that the assistance is minor enough to not warrant co-authorship. This levels the playing field for human and AI assistance, as prominent journals have already expressed that LLMs cannot be listed as co-authors<ref name=":0" /> as had already been done in some papers.<ref name=":2">{{Cite journal |last=Kung |first=Tiffany H. |last2=Cheatham |first2=Morgan |last3=ChatGPT |last4=Medenilla |first4=Arielle |last5=Sillos |first5=Czarina |last6=De Leon |first6=Lorie |last7=Elepaño |first7=Camille |last8=Madriaga |first8=Maria |last9=Aggabao |first9=Rimel |last10=Diaz-Candido |first10=Giezel |last11=Maningo |first11=James |date=2022-12-20 |title=Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models |url=http://medrxiv.org/lookup/doi/10.1101/2022.12.19.22283643 |language=en |doi=10.1101/2022.12.19.22283643}}</ref>

Second, we examined in which aspects of the research process are the prospective human-AI disparities the strongest. If—as we hypothesize—delegating to AI is perceived less favorably, then one can assume that delegating such processes to AI will have the greatest potential to devalue work done by scientists.

===Participants===
To assess the consequences of delegating research processes to LLMs, 441 participants from Prolific<ref>{{Cite journal |last=Palan |first=Stefan |last2=Schitter |first2=Christian |date=2018-03 |title=Prolific.ac—A subject pool for online experiments |url=https://linkinghub.elsevier.com/retrieve/pii/S2214635017300989 |journal=Journal of Behavioral and Experimental Finance |language=en |volume=17 |pages=22–27 |doi=10.1016/j.jbef.2017.12.004}}</ref> were recruited. Prolific is an online crowdsourcing platform used to collect primary data from humans, including experimental data.<ref>{{Cite journal |last=Peer |first=Eyal |last2=Brandimarte |first2=Laura |last3=Samat |first3=Sonam |last4=Acquisti |first4=Alessandro |date=2017-05 |title=Beyond the Turk: Alternative platforms for crowdsourcing behavioral research |url=https://linkinghub.elsevier.com/retrieve/pii/S0022103116303201 |journal=Journal of Experimental Social Psychology |language=en |volume=70 |pages=153–163 |doi=10.1016/j.jesp.2017.01.006}}</ref> For a long time, Amazon Mechanical Turk was the dominant online labor market, i.e., a marketplace where individuals can complete tasks—such as participate in a research study—for compensation.<ref>{{Cite journal |last=Buhrmester |first=Michael |last2=Kwang |first2=Tracy |last3=Gosling |first3=Samuel D. |date=2011-01 |title=Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? |url=http://journals.sagepub.com/doi/10.1177/1745691610393980 |journal=Perspectives on Psychological Science |language=en |volume=6 |issue=1 |pages=3–5 |doi=10.1177/1745691610393980 |issn=1745-6916}}</ref> However, our experience, as well as some research, has shown that data gathered using Prolific is superior<ref>{{Cite journal |last=Peer |first=Eyal |last2=Rothschild |first2=David |last3=Gordon |first3=Andrew |last4=Evernden |first4=Zak |last5=Damer |first5=Ekaterina |date=2021-09-29 |title=Data quality of platforms and panels for online behavioral research |url=https://link.springer.com/10.3758/s13428-021-01694-3 |journal=Behavior Research Methods |language=en |volume=54 |issue=4 |pages=1643–1662 |doi=10.3758/s13428-021-01694-3 |issn=1554-3528 |pmc=PMC8480459 |pmid=34590289}}</ref>, and thus we decided to use this platform. To further ensure a high quality of data and a relatively homogenous sample, we recruited participants who had a 98% or higher approval rating, were located and born in the United States, and whose first language was English. As preregistered, thirty-nine participants that did not correctly answer both attention check questions were excluded, leaving a final sample size of 402 (48.3% female, 49.8% male, and 1.9% selected non-binary or did not disclose). The mean age of participants was 42.0 years (''SD'' = 13.9). Of the final sample size, 97.5% had heard about ChatGPT, and 38.1% interacted with it.

The study was pre-registered at https://aspredicted.org/GVL_MR5. Data and materials are available at https://osf.io/fsavc/. The data file includes a short description of all variables used in the analysis.

===Experimental design===
We conducted a mixed-design experiment. We randomly allocated participants to one of two conditions between-subjects. Participants rated a distinguished senior researcher who delegated a part of the research process to either another person—specifically, a PhD student with two years’ experience in the area (human condition), or to an LLM such as ChatGPT (LLM condition). Each participant rated the effect of such delegation on each of the five parts of the research process discussed in Cargill and O’Connor<ref name=":3">{{Cite book |last=Cargill |first=Margaret |last2=O'Connor |first2=Patrick |date=2021 |title=Writing scientific research articles: strategy and steps |edition=Third edition |publisher=Wiley-Blackwell |place=Hoboken, NJ |isbn=978-1-119-71727-0}}</ref>: idea generation, prior literature synthesis, data identification and preparation, testing framework determination and implementation, and results analysis. Notably, Dowling and Lucey<ref name=":1" /> used all of these except results analysis to assess the quality of ChatGPT’s output. We rephrased the two last research processes for clarity.

For each research process, the participants rated the extent to which they agreed with three items, on a Likert scale of 1 (strongly disagree) to 7 (strongly agree):

*I think that it is morally acceptable for a scientist to delegate—in such a scenario—the following part of the research process (after giving credit in the acknowledgments);
*I think that a scientist that delegated the part of the research process shown below should be trusted to oversee future research projects; and
*I think that delegating this part of the research process will produce correct output and stand up to scientific scrutiny (e.g., results would be robust, reliable, and correctly interpreted).

We expected the first two items to correlate with one another but not necessarily with the third. While people might acknowledge that AI might be better than humans in some tasks, they often exhibit an aversion toward the use of algorithms.<ref>{{Cite journal |last=Dietvorst |first=Berkeley J. |last2=Simmons |first2=Joseph P. |last3=Massey |first3=Cade |date=2015 |title=Algorithm aversion: People erroneously avoid algorithms after seeing them err. |url=https://doi.apa.org/doi/10.1037/xge0000033 |journal=Journal of Experimental Psychology: General |language=en |volume=144 |issue=1 |pages=114–126 |doi=10.1037/xge0000033 |issn=1939-2222}}</ref>

Given that each participant rated three different items for five different research processes, we obtained fifteen data points per participant. The main analysis (see Table 2, in the next section) is performed on various levels: the pooled dataset (with 15 data points per participant), and separately for: (1) each of the three items, and (2) each of the five research processes.

==Results==
===Preliminary analysis===
Prior to presenting the regression results, we examined as to how answers correlated with each other. As expected, moral acceptability ratings (with correlations based on mean ratings from the five research processes) correlated highly with trust to oversee future projects, ''r'' = 0.81, ''p'' < 0.001. However, moral acceptability ratings also correlated highly with accuracy ratings, ''r'' = 0.81, ''p'' < 0.001. Similarly, trust ratings correlated highly with accuracy, ''r'' = 0.80, ''p'' < 0.001.

However, it remains possible that the relationship between such perceptions was lower when the scientist delegated to an LLM instead of a human. To determine this, we conducted a regression analysis treating one item as the dependent variable, and another as the independent variable, but we added an interaction with a dummy variable across delegation condition. Results, presented in Table 1, suggest that the strength of the relationship between moral acceptability, trust, and accuracy either becomes stronger when delegating to an LLM (rather than a human) or is not statistically different. Therefore, people evaluated moral acceptability, trust, and accuracy in a similar manner in each condition.

{| 
 | style="vertical-align:top;" |
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="60%"
 |-
  | colspan="4" style="background-color:white; padding-left:10px; padding-right:10px;" |'''Table 1.''' The interrelationship between ratings of three items (moral acceptability, trust to oversee, and accuracy). Notes: Ratings are means for five research processes. Moral acceptability, trust, and accuracy scores are standardized to facilitate the interpretability of the coefficient for LLM (which corresponds to the effect of delegating to the LLM [relative to the human] when trust or accuracy is at its mean level). * p < 0.05; ** p < 0.01; *** p < 0.001.
 |- 
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Moral acceptability
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Trust
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Accuracy
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |(Intercept)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.12*<br />(0.05)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.11*<br />(0.05)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.10<br />(0.05)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Trust
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.68***<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |LLM (1 = yes, 0 = no)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.17**<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.15*<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.16*<br />(0.07)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Trust ∙ LLM
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.14*<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Accuracy
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.67***<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.72***<br />(0.06)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Accuracy ∙ LLM
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.16*<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.07<br />(0.07)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''N''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''R''<sup>2</sup> adjusted
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.660
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.656
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.636
 |-
|}
|}

===Pre-registered analysis===
We present the results of the pre-registered analysis in Table 2. Consistent with the hypothesis, people rated delegating the research process to an LLM as less morally acceptable and reported lower trust towards this scientist to oversee future research projects. Moreover, people also rated delegating to an LLM as producing less correct output. The effect of delegating to an LLM (relative to delegating the same to a PhD student) was similar for all three items, and thus results from the combined dataset (“All items and processes”) can serve as a benchmark for future studies.

For readers accustomed to Cohen’s ''d''<ref name=":4">{{Cite book |last=Cohen |first=Jacob |date=1988 |title=Statistical power analysis for the behavioral sciences |edition=2nd ed |publisher=L. Erlbaum Associates |place=Hillsdale, N.J |isbn=978-0-8058-0283-2}}</ref>, the effect sizes (and 95% confidence intervals) of delegating to an LLM instead of a human were large: ''d'' = –0.78 [–0.99, –0.58] for moral acceptability, ''d'' = –0.80 [–1.00, –0.60] for trust, and ''d'' = –0.85 [–1.06, –0.65] for accuracy.

{| 
 | style="vertical-align:top;" |
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="70%"
 |-
  | colspan="10" style="background-color:white; padding-left:10px; padding-right:10px;" |'''Table 2.''' Perceptions of delegating parts of the research process. Notes: Linear mixed effect models were estimated using [[R (programming language)|R]] packages lme4<ref>{{Cite journal |last=Bates |first=Douglas |last2=Mächler |first2=Martin |last3=Bolker |first3=Ben |last4=Walker |first4=Steve |date=2015 |title=Fitting Linear Mixed-Effects Models Using lme4 |url=http://www.jstatsoft.org/v67/i01/ |journal=Journal of Statistical Software |language=en |volume=67 |issue=1 |doi=10.18637/jss.v067.i01 |issn=1548-7660}}</ref> and lmerTest.<ref>{{Cite journal |last=Kuznetsova |first=Alexandra |last2=Brockhoff |first2=Per B. |last3=Christensen |first3=Rune H. B. |date=2017 |title=lmerTest Package: Tests in Linear Mixed Effects Models |url=http://www.jstatsoft.org/v82/i13/ |journal=Journal of Statistical Software |language=en |volume=82 |issue=13 |doi=10.18637/jss.v082.i13 |issn=1548-7660}}</ref> The baseline values are “Moral acceptability” for Item, “Idea generation” for Research process, and "female" for gender. All variables bar age and gender are dummy variables, taking the value "1" if the variable is equal to what the variable’s name implies, and "0" otherwise. * p < 0.05; ** p < 0.01;   *** p < 0.001.
 |- 
  ! rowspan="2" style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |
  ! rowspan="2" style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |All items and processes
  ! colspan="3" style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Items
  ! colspan="5" style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Research processes
 |- 
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Moral acceptability
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Trust
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Correctness
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Idea generation
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Prior literature synthesis
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Data identification and preparation
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Testing and interpreting the theoretical framework
  ! style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Statistical result analysis
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |(Intercept)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |5.34***<br />(0.09)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |4.92***<br />(0.46)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |4.90***<br />(0.50)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |5.16***<br />(0.46)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |4.83***<br />(0.53)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |5.24***<br />(0.48)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |5.34***<br />(0.48)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |5.52***<br />(0.53)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |4.67***<br />(0.51)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |LLM
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.07***<br />(0.12)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.01***<br />(0.13)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.13***<br />(0.14)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.09***<br />(0.13)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.89***<br />(0.15)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.12***<br />(0.14)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.23***<br />(0.13)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–1.30***<br />(0.15)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.85***<br />(0.14)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Item = ''Correctness''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.16***<br />(0.03)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.08<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.16**<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.25***<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.07<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.22***<br />(0.06)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Item = ''Trust''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.07*<br />(0.03)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.11<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.04<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.05<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.20***<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.02<br />(0.06)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Research process = ''Prior literature synthesis''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.21***<br />(0.04)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.26***<br />(0.08)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.19**<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.18**<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Research process = ''Data identification and preparation''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.28***<br />(0.04)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.39***<br />(0.08)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.23***<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.22**<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Research process = ''Testing and interpreting the theoretical framework''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.11**<br />(0.04)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.15<br />(0.08)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.05<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.14*<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Research process = ''Statistical result analysis''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.11*<br />(0.04)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.18*<br />(0.08)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.09<br />(0.06)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.05<br />(0.07)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Age
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01<br />(0.00)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01<br />(0.01)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01<br />(0.00)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.00<br />(0.01)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01<br />(0.00)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01*<br />(0.00)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.00<br />(0.01)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.02**<br />(0.01)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Heard of ChatGPT
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.01<br />(0.42)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.14<br />(0.45)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.23<br />(0.41)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.26<br />(0.48)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.09<br />(0.44)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.10<br />(0.43)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.38<br />(0.48)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.00<br />(0.47)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Interacted with ChatGPT
  | style="background-color:white; padding-left:10px; padding-right:10px;" |
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.02<br />(0.13)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.06<br />(0.14)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.12<br />(0.13)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.26<br />(0.16)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.00<br />(0.14)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.01<br />(0.14)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.12<br />(0.16)
  | style="background-color:white; padding-left:10px; padding-right:10px;" |–0.19<br />(0.15)
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Gender
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
  | style="background-color:white; padding-left:10px; padding-right:10px;" |Yes
 |-
  ! colspan="10" style="background-color:#e2e2e2; padding-left:10px; padding-right:10px;" |Random effects
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''σ''<sup>2</sup>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.12
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.20
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.77
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.93
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.79
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.64
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.64
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.70
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.71
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''τ''<sub>00</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.40<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.41<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.78<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.45<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.96<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.62<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.58<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.99<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1.84<sub>id</sub>
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''ICC''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.56
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.54
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.70
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.61
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.71
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.72
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.71
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.74
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.72
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''N''<sub>participants</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |402<sub>id</sub>
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''N''
  | style="background-color:white; padding-left:10px; padding-right:10px;" |6030
  | style="background-color:white; padding-left:10px; padding-right:10px;" |2010
  | style="background-color:white; padding-left:10px; padding-right:10px;" |2010
  | style="background-color:white; padding-left:10px; padding-right:10px;" |2010
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1206
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1206
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1206
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1206
  | style="background-color:white; padding-left:10px; padding-right:10px;" |1206
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |''Marginal R''<sup>2</sup>/<br />''Conditional R''<sup>2</sup>
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.111 / 0.606
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.106 / 0.589
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.119 / 0.733
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.127 / 0.660
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.078 / 0.736
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.131 / 0.752
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.162 / 0.757
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.151 / 0.779
  | style="background-color:white; padding-left:10px; padding-right:10px;" |0.096 / 0.748
 |-
|}
|}

===Exploratory analysis===
Table 2 (above) and Figure 1 present how ratings varied across the five research processes and conditions. The adverse effect of delegating to an LLM was strongest for the “Testing and interpreting the theoretical framework” process and weakest for the “Statistical result analysis” process. However, the patterns were robust for each of the five research elements, in the ''d'' = –0.81 (large effect) to –0.51 (medium effect) range. Therefore, despite some variation across research processes, people nonetheless judged delegation of any process to an LLM as worse than to a human.


[[File:Fig1 Niszczota EconBusRev23 9-2.png|700px]]
{{clear}}
{| 
 | style="vertical-align:top;" |
{| border="0" cellpadding="5" cellspacing="0" width="700px"
 |-
  | style="background-color:white; padding-left:10px; padding-right:10px;" |<blockquote>'''Figure 1.''' Ratings of moral acceptability, trustworthiness of scientist to oversee future projects after delegation, and accuracy of science across research processes. Dots represent means.</blockquote>
 |- 
|}
|}

==Discussion==
Overall, these results suggest that people have clear, strong negative views of scientists delegating any aspect of the research process to ChatGPT or similar LLMs, compared to a PhD student. They rated delegation to an LLM as less morally acceptable, a scientist choosing such delegation as less trustworthy for future projects, and they rated the output of such delegation as less accurate and of lower quality. These ratings held across all five aspects of the research process identified in past work: idea generation, prior literature synthesis, data identification and preparation, testing framework determination and implementation, and results analysis.<ref name=":3" /> Although people showed the strongest differentiation between LLMs and human researchers for testing and interpreting the theoretical framework, and the weakest for statistical result analysis, the effect size for all five was substantial, with Cohen’s ''d''’s that would be conventionally described as medium to large<ref name=":4" />, but can be considered large to very large<ref>{{Cite journal |last=Funder |first=David C. |last2=Ozer |first2=Daniel J. |date=2019-06 |title=Evaluating Effect Size in Psychological Research: Sense and Nonsense |url=http://journals.sagepub.com/doi/10.1177/2515245919847202 |journal=Advances in Methods and Practices in Psychological Science |language=en |volume=2 |issue=2 |pages=156–168 |doi=10.1177/2515245919847202 |issn=2515-2459}}</ref> based on effect sizes that are observed in psychological research.

Note that, as expected, moral ratings and trust in scientists were highly correlated, but additionally both correlated highly with perceptions of accuracy and scientific quality. One possibility for this pattern is that people think that delegating to LLMs is immoral and untrustworthy precisely because they view the output of such programs as scientifically questionable. This pattern leaves open the possibility that, with further advancement in AI, if the perceived scientific quality of LLMs increases, people may view delegation to such programs as less problematic.

Nonetheless, these results have clear implications for researchers considering use of ChatGPT or other LLMs. At least in their current state, people view such delegation as seriously problematic—as immoral, untrustworthy, and scientifically unsound. This view extends to all aspects of the research process. Therefore, there does not appear to be a widely approved way for researchers to incorporate LLMs into the research process without compromising their work’s perceived quality and integrity.

It is worth noting that the current work examined the case where the researcher honestly reports the use of the LLM in the acknowledgments section, as recommended by leading journals such as ''Science''.<ref name=":0" /> Moreover, the current work examined the case where the researcher delegating to a PhD student—essentially the control condition—features them only in the acknowledgments section rather than as a co-author. Arguably, people may view doing so as ethically questionable as the graduate student would have earned authorship according to common ethical guidelines such as those published by the American Psychological Association.<ref>{{Cite book |date=2020 |editor-last=American Psychological Association (Washington, District of Columbia) |title=Publication manual of the American psychological association |edition=Seventh edition |publisher=American Psychological Association |place=Washington, DC |isbn=978-1-4338-3215-4}}</ref> Therefore, the current findings represent a plausible best-case scenario—it is plausible that people would have even stronger negative reactions to a researcher who employed LLMs without revealing their use and who essentially takes credit for the output of an algorithm as compared to a researcher giving their PhD student colleague full authorship credit. These findings underscore the depth of the antipathy toward researchers using LLMs at this time.

===Limitations===
As with all studies, the current work suffers from some limitations. First, we compared delegation to LLMs to a second-year PhD student, a human with presumably sufficient competence as to normally warrant authorship in scientific publications. Naturally, the choice of comparison target should affect responses. For example, people may think that LLMs will produce more accurate output than, say, a four-year-old or someone who is illiterate. Future work could plausibly test how people perceive LLMs compared to a wide range of targets. However, we elected to begin by testing LLMs against someone who would likely otherwise participate in the scientific process.

Second, we examined the perceptions of laypeople who may have only vague familiarity or understanding of the scientific process. It remains to be seen whether journal editors, reviewers, senior university officials, and others who intimately understand the research process and evaluate scientists share the same views. It may be that with such familiarity, people perceive it more permissible to use LLMs for specific aspects such as [[data analysis]]. Findings might also differ using a different split of the research process, perhaps one that includes more fine-grained elements like generating figures based on data computed by humans.<ref name=":1" /><ref name=":3" />

Likewise, we examined only perceptions of a scientist operating in a particular area, namely a researcher specializing in economics, finance, and psychology. It remains possible that people hold less-negative views of LLM usage in other branches of science, e.g., perhaps for papers in astrophysics requiring complex calculations. Along the same lines, results may be moderated by the perceived goals of the scientist; for example, it seems likely that people would not hold the same negative impression of research specifically designed to illustrate the uses and limitations of ChatGPT itself.<ref name=":2" />

Moreover, we asked people about a hypothetical scientist. It remains possible that asking about a specific (e.g., famous, eminent, trusted) scientist people demonstrate a lower aversion to LLM use, perhaps because they may infer this trusted scientist would only use LLMs if they had specialist knowledge that doing so was worthwhile and not likely to corrupt the research process. In other words, people may moderate inferences about the use of LLMs depending on their prior knowledge and evaluations of a specific scientist.

Finally, the current work examined American participants. It remains possible that results may vary in other populations; for example, Americans tend to view AI more critically than people in China.<ref>{{Cite journal |last=Wu |first=Yuheng |last2=Mou |first2=Yi |last3=Li |first3=Zhipeng |last4=Xu |first4=Kun |date=2020-03 |title=Investigating American and Chinese Subjects’ explicit and implicit perceptions of AI-Generated artistic work |url=https://linkinghub.elsevier.com/retrieve/pii/S074756321930398X |journal=Computers in Human Behavior |language=en |volume=104 |pages=106186 |doi=10.1016/j.chb.2019.106186}}</ref> Furthermore, not all scientists have equal access to state-of-the-art language models. For example, people from China cannot access these models<ref>{{Cite journal |last=Wang |first=Simon H. |date=2023-03-02 |title=OpenAI — explain why some countries are excluded from ChatGPT |url=https://www.nature.com/articles/d41586-023-00553-9 |journal=Nature |language=en |volume=615 |issue=7950 |pages=34–34 |doi=10.1038/d41586-023-00553-9 |issn=0028-0836}}</ref>, and Italy has banned access to ChatGPT, at least temporarily.<ref>{{Cite web |last=Satariano, A. |date=31 March 2023 |title=ChatGPT Is Banned in Italy Over Privacy Concerns |work=The New York Times |url=https://www.nytimes.com/2023/03/31/technology/chatgpt-italy-ban.html}}</ref> As such, the perceptions of research delegated to LLM may vary somewhat with access to such models or which models are popular or available.

==Conclusions==
Overall, the current findings suggest that people have strongly negative views of delegating any aspect of the research process to LLMs such as ChatGPT, compared to a junior human scientist: people rated doing so more immoral, more untrustworthy, and the results as less accurate and of lower quality. These findings held for five aspects of the research process, from idea generation to data analysis. Therefore, researchers should employ caution when considering whether to incorporate ChatGPT or other LLMs into their research. It appears that even when disclosing such practices according to modern standards, doing so may powerfully reduce perceptions of scientific quality and integrity.

==Acknowledgements==
===Funding===
This research was supported by grant 2021/42/E/HS4/00289 from the National Science Centre, Poland.

===Conflict of interest===
None stated.

==References==
{{Reflist|colwidth=30em}}

==Notes==
This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The original lists references in alphabetical order; this version lists them in order of appearance, by design.

<!--Place all category tags here-->
[[Category:LIMSwiki journal articles (added in 2024)]]
[[Category:LIMSwiki journal articles (all)]]
[[Category:LIMSwiki journal articles on artificial intelligence]]
[[Category:LIMSwiki journal articles on research]]