Supporting High-Level to Low-Level Requirements Coverage Reviewing with Large Language Models
Sprache des Vortragstitels:
Englisch
Original Tagungtitel:
21st International Conference on Mining Software Repositories (MSR ?24), April , 2024, Lisbon, Portugal.
Sprache des Tagungstitel:
Englisch
Original Kurzfassung:
Refining high-level requirements into low-level ones is a common
task, especially in safety-critical systems engineering. The objective
is to describe every important aspect of the high-level requirement
in a low-level requirement, ensuring a complete and correct im-
plementation of the system?s features. To this end, standards and
regulations for safety-critical systems require reviewing the cover-
age of high-level requirements by all its low-level requirements to
ensure no missing aspects.
The challenge of supporting automatic reviews for requirements
coverage originates from the distinct levels of abstraction between
high-level and low-level requirements, their reliance on natural
language, and the often different vocabulary used. The rise of Large
Language Models (LLMs), trained on extensive text corpora and ca-
pable of contextualizing both high-level and low-level requirements,
opens new avenues for addressing this challenge.
This paper presents an initial study to explore the performance
of LLMs in assessing requirements coverage. We employed GPT-3.5
and GPT-4 to analyze requirements from five publicly accessible
data sets, determining their ability to detect if low-level require-
ments sufficiently address the corresponding high-level require-
ment. Our findings reveal that GPT-3.5, utilizing a zero-shot prompt-
ing strategy augmented with the prompt of explaining, correctly
identifies complete coverage in four out of five evaluation data
sets. Additionally, it exhibits an impressive 99.7% recall rate in ac-
curately identifying instances where coverage is incomplete due
to removing a single low-level requirement across our entire set of
evaluation data.