Browsing by Author "BATISTA, ANA ISABEL MOURA"
Now showing 1 - 1 of 1
Results Per Page
Sort Options
- Assessing the effectiveness of Large Language Models in automated threat modelingPublication . BATISTA, ANA ISABEL MOURA; Pereira, Nuno Alexandre MagalhãesAs cyberattacks become more frequent, Threat Modeling has emerged as an essential component of software security practices. Traditionally, Threat Modeling is an intensive process, relying on experts to identify and evaluate risks within a system, which limits its adoption. The advent of Large Language Models (LLMs) presents an opportunity to automate this process. However, the successful application of these models in Threat Modeling requires careful prompt engineering and a rigorous strategy to assess the generated threat scenarios. The project investigates this applicability, centering on a case study involving the Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI) SUNDIAL application. Using STRIDE GPT as the tool for threat models generation, four prompting techniques were studied and applied: STRIDE GPT’s Initial Prompt, Chain of Thought (CoT), Negative-Only Few-Shot, and the combined NO-Few-Shot-CoT, across three LLMs. A Threat Model Evaluation Tool, TMEval, is proposed to enable focused comparison of identified STRIDE threats by LLMs against those in the ground truth for a specific application, employing four metrics: BLEU, ROUGE, BERTScore, and LLM-as-a-Judge. The emphasis is on the LLM-as-a-Judge approach across five dimensions: consistency, plausibility, and coverage of targets, weaknesses, and attack vectors. The results show that any LLM with a specific prompting strategy does not produce scenarios consistent with the ground truth across all threat categories, suggesting that performance depends on the category and the application context provided. For the case study, the NOFew- Shot-CoT prompting approach demonstrated the highest effectiveness across most categories.