A carregar...
Publicação
“Can a chatbot be used in the full-text screening in a systematic review?”
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 3.04 MB | Adobe PDF |
Orientador(es)
Resumo(s)
Large language model–based artificial intelligence tools are increasingly explored to support systematic reviews, yet evidence regarding their reliability in full-text screening remains limited. This study evaluated the performance of two versions of ChatGPT (4.0 and 5.0) compared with human reviewers during article selection for a systematic review on influenza vaccine effectiveness. A total of 170 full-text articles were independently assessed for eligibility using predefined inclusion and exclusion criteria. Human reviewers served as the gold standard. ChatGPT 4.0 and 5.0 were prompted using standardized instructions mirroring the review protocol. Agreement with human decisions was evaluated using accuracy, sensitivity, specificity, precision, F1-score, and Cohen’s κ. Intra-model reproducibility was assessed for ChatGPT 5.0. Results: ChatGPT 4.0 achieved an accuracy of 0.71 (95% CI: 0.64–0.78) and a Cohen’s κ of 0.43, indicating moderate agreement with human reviewers. ChatGPT 5.0 demonstrated improved performance, with accuracy increasing 0.06 to 0.77 (95% CI: 0.70–0.83), sensitivity of 0.87, specificity of 0.70, and κ of 0.55, corresponding to moderate-to-substantial agreement. Intra-model reproducibility for ChatGPT 5.0 showed 80% agreement (κ = 0.60), indicating partial but imperfect consistency. ChatGPT 5.0 outperformed ChatGPT 4.0 in full-text screening accuracy and reproducibility, approaching but not matching human performance. These findings support the use of current LLMs as decision- support tools rather than autonomous reviewers in systematic reviews. Transparent reporting of model versions, prompts, and input quality is essential to ensure credible AI-assisted evidence synthesis.
Descrição
Palavras-chave
Systematic review ChatGPT Artificial intelligence Large language models Full-text screening Reproducibility Chatbot
Contexto Educativo
Citação
Martins, A. M., Valero Juan, L. F., Oliveira, A., Martins, J. P., & Santos, M. (2026). “Can a chatbot be used in the full-text screening in a systematic review?” International Journal of Medical Informatics, 214, 106422. https://doi.org/10.1016/j.ijmedinf.2026.106422
Editora
Elsevier