Loading...
Publication
IA na geração de relatórios técnico-científicos de apoio à decisão em farmacologia clínica
| Name: | Description: | Size: | Format: | |
|---|---|---|---|---|
| 5.83 MB | Adobe PDF |
Authors
Advisor(s)
Abstract(s)
Esta dissertação apresenta uma prova de conceito (PoC) para a aplicação de técnicas de IAG, em particular Large Language Models, na automatização da elaboração de relatórios técnicos no domínio da Farmacologia Clínica (FC). O trabalho tem como motivação apoiar a Unidade de Farmacologia Clínica (UFC) do Hospital de São João, Porto, na produção de relatórios consistentes e cientificamente fundamentados para responder a pedidos clínicos complexos (ex.: implementação de terapêuticas off-label, avaliação do custo-benefício de um fármaco em
determinado contexto clínico, utilização de um medicamento em detrimento de outro, entre outros), pedidos estes realizados pela Comissão de Farmácia e Terapêutica (CFT). O sistema proposto integra metodologias como Retrieval-Augmented Generation (RAG), embeddings semânticos e servidores MCP, procurando replicar, com a maior fidelidade possível, os processos seguidos pela UFC. A investigação centrou-se em dois eixos principais: a recuperação de informação a partir de documentos regulamentares médicos, sobretudo, o “Resumo das Características do Medicamento” (RCM) dos fármacos, através de pesquisa semântica baseada em embeddings;
e a revisão automatizada de literatura, recorrendo a um servidor MCP (BioMCP) para recuperar e sintetizar evidência relevante do PubMed e de fontes complementares. Diversos modelos de embedding e LLMs foram avaliados de forma sistemática, utilizando datasets especificamente construídos para o caso, gold standards validados por especialistas em farmacologia e métricas clássicas de recuperação de informação (MAP, MRR, NDCG, Recall, Precision, entre outras). Os resultados evidenciam o desempenho superior do text-embedding-ada-002 da OpenAI na ordenação de documentos e do GPT-5 na estabilidade da pesquisa bibliográfica e na geração de relatórios, embora persistam limitações quanto à robustez, adequação clínica e consistência.
O protótipo desenvolvido em Streamlit operacionaliza todo o processo, desde a receção estruturada do pedido (título, enquadramento e caso clínico) até à construção modular do relatório. As estratégias de avaliação combinaram análise de cobertura lexical, utilização de LLM-as-a-judge e aplicação de checklists estruturadas. Embora os resultados obtidos pelo sistema demonstrem a viabilidade técnica e o enorme potencial da automação baseada em IA neste processo em específico, os resultados sublinham que ainda são necessárias bastantes melhorias antes de sua adoção numa área científica tão crítica como a farmacologia/biomédica.
Trabalhos futuros deverão centrar-se no fine-tuning de modelos específicos para português na área biomédica, na integração de arquiteturas híbridas de recuperação, em pipelines de validação mais robustos, em mecanismos de explicabilidade e na inclusão de especialistas humanos no processo de geração dos relatórios, algo que será crítico em fases posteriores. Em última análise, esta investigação contribui para definir padrões de conceção e metodologias de avaliação da aplicação de IA generativa em contextos médicos de elevada responsabilidade,
abrindo caminho para sistemas de apoio à decisão clínica fiáveis e seguros.
This dissertation presents a proof of concept (PoC) for the application of Generative Artificial Intelligence techniques, particularly Large Language Models (LLMs), in the automation of technical report generation in the field of Clinical Pharmacology. The work is motivated by the need to support the Clinical Pharmacology Unit (UFC) of Hospital de São João, Porto, in producing timely, consistent, and scientifically grounded reports for complex clinical requests mad by the Pharmacy and Therapeutics Commission (CFT). The proposed system integrates methodologies such as Retrieval-Augmented Generation (RAG), semantic embeddings and MCP servers to replicate, as closely as possible, the processes followed by the hospital’s Clinical Pharmacology Unit. The research focused on two main pillars: information retrieval from regulatory documents, namely the Summary of Product Characteristics (SmPC), through embedding-based semantic search; and automated literature review, leveraging a biomedical MCP server (BioMCP) to retrieve and synthesize relevant evidence from PubMed and complementary sources. Multiple embedding models and LLMs were systematically evaluated using purpose-built datasets, gold standards validated by pharmacology experts, and classical IR metrics (MAP, MRR, NDCG, Recall, Precision). The results highlight the superior performance of OpenAI’s text-embeddingada-002 in document ranking and GPT-5 in literature search stability and report generation, although limitations remain regarding robustness, clinical adequacy, and consistency. The developed Streamlit-based prototype operationalizes the full pipeline, from structured input (title, context, clinical case) to the modular construction of a draft report. Evaluation strategies combined lexical coverage analysis, LLM-as-a-judge assessments, and structured checklists. While the system demonstrates the technical feasibility and potential of AI-driven automation in Clinical Pharmacology, the findings highlight that substantial improvements are still required before their full adoption in such a critical scientific domain as pharmacology/biomedicine. Future work should address fine-tuning of domain-specific models in European Portuguese, integration of hybrid retrieval architectures, enhanced validation pipelines, explainability mechanisms and human in the loop within the report generation process. Ultimately, this research contributes to defining design patterns and evaluation methodologies for applying generative AI to high-stakes medical contexts, paving the way for trustworthy clinical decision support systems.
This dissertation presents a proof of concept (PoC) for the application of Generative Artificial Intelligence techniques, particularly Large Language Models (LLMs), in the automation of technical report generation in the field of Clinical Pharmacology. The work is motivated by the need to support the Clinical Pharmacology Unit (UFC) of Hospital de São João, Porto, in producing timely, consistent, and scientifically grounded reports for complex clinical requests mad by the Pharmacy and Therapeutics Commission (CFT). The proposed system integrates methodologies such as Retrieval-Augmented Generation (RAG), semantic embeddings and MCP servers to replicate, as closely as possible, the processes followed by the hospital’s Clinical Pharmacology Unit. The research focused on two main pillars: information retrieval from regulatory documents, namely the Summary of Product Characteristics (SmPC), through embedding-based semantic search; and automated literature review, leveraging a biomedical MCP server (BioMCP) to retrieve and synthesize relevant evidence from PubMed and complementary sources. Multiple embedding models and LLMs were systematically evaluated using purpose-built datasets, gold standards validated by pharmacology experts, and classical IR metrics (MAP, MRR, NDCG, Recall, Precision). The results highlight the superior performance of OpenAI’s text-embeddingada-002 in document ranking and GPT-5 in literature search stability and report generation, although limitations remain regarding robustness, clinical adequacy, and consistency. The developed Streamlit-based prototype operationalizes the full pipeline, from structured input (title, context, clinical case) to the modular construction of a draft report. Evaluation strategies combined lexical coverage analysis, LLM-as-a-judge assessments, and structured checklists. While the system demonstrates the technical feasibility and potential of AI-driven automation in Clinical Pharmacology, the findings highlight that substantial improvements are still required before their full adoption in such a critical scientific domain as pharmacology/biomedicine. Future work should address fine-tuning of domain-specific models in European Portuguese, integration of hybrid retrieval architectures, enhanced validation pipelines, explainability mechanisms and human in the loop within the report generation process. Ultimately, this research contributes to defining design patterns and evaluation methodologies for applying generative AI to high-stakes medical contexts, paving the way for trustworthy clinical decision support systems.
Description
Keywords
Large Language Models Retrieval Augmented Generation Model Context Protocol Clinical Pharmacology Proof of Concept Farmacologia Clínica Prova de conceito