ANALYSIS ON LARGE LANGUAGE MODEL VULNERABLE CODE GENERATION AND SELF-REPAIR ABILITY

Abstract

This thesis investigates Large Language Models' (LLMs) propensity to produce vulnerable code and their self-repair capabilities in coding. Analyzing a novel dataset from real-world prompts, including 751 instances of vulnerable code generated from 90 prompts by ChatGPT, the study employs Static Application Security Testing tools to examine these issues. It introduces two strategies for reducing vulnerabilities: "iteration repair," which iteratively corrects generated code, and "preshot repair," anticipating vulnerabilities to prevent insecure code generation. Implemented in "Codexity," a tool with a VS Code extension, these methods significantly reduced vulnerable code production, with "iteration repair" achieving a 60% reduction and "preshot repair" up to 36.5%. The effectiveness of these strategies is highlighted through comparisons with existing tools, demonstrating LLMs' potential to improve coding security and efficiency.