Semantic-Aware LLM-Based Test Generation: A Closed-Loop Framework for Requirement Alignment and Fault Detection

Abstract: Large language models (LLMs) increasingly generate software tests from source codes, natural language requirements, and API specifications. While LLMs can produce readable test cases with plausible assertions, the generated tests may fail to validate requirement intent, omit boundary conditions, duplicate scenarios, or provide weak fault detection. This study proposes a semantic-aware closed-loop framework for LLM-based test generation. The framework combines requirement ingestion, retrieval-augmented context construction, constraint extraction, LLM-based test generation, semantic coverage scoring, mutation-style feedback and refinement. The main contribution is a Semantic Coverage Metric (SCM) that evaluates requirement-test alignment using semantic similarity, constraint satisfaction, and assertion quality. To strengthen the framework beyond conceptual design, this study includes an executable pilot study using StringUtils-style utility functions, repeated generated test suites, coverage analysis, and mutation-style fault injection. The pilot study shows that semantic-guided generation improves line coverage and mutation effectiveness compared with naive prompting and slightly outperforms structured prompting in coverage while maintaining explicit requirement traceability. This study demonstrates that LLM-generated tests should be evaluated using both traditional quality metrics and requirement-alignment measures before adoption in engineering workflows.

Keywords: LLM-generated tests, software testing, semantic coverage, mutation testing, requirement traceability, AI- assisted testing, test generation, software quality.